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Abstract 

In  support  of  senior  leadership  emphasis  on  improving  early  systems  engineering  and 
analysis,  the  Enterprise  Requirements  and  Acquisition  Model  (ERAM)  is  a  quantitative  discrete- 
event  process  simulation  model  accounting  for  activities  from  early  capability  analysis  through 
system  fielding.  The  model  begins  with  Air  Force  Space  Command’s  (AFSPC)  identification  of 
a  desired  space  capability  early  in  the  Joint  Capabilities  and  Integration  Development  System 
(JCIDS)  process  through  system  development  at  Milestone-C  (MS-C)  of  the  acquisition  system 
resulting  in  a  probabilistic  schedule  distribution  for  a  given  concept.  This  model  of  the 
Department  of  Defense's  (DoD)  space  capability  development  process  provides  decision  making 
information  for  the  Concept  Characterization  and  Technical  Description  (CCTD)  documents 
referenced  during  Analysis  of  Alternatives  (AoA).  The  research  focused  on  identifying  activities 
and  assigning  historical  distributions  and  probabilities  at  each  decision  point.  Data  was  collected 
through  analysis  of  applicable  policy,  instructions,  and  journal  articles  as  well  as  interviews  with 
subject  matter  experts  (SME)  from  the  Air  Staff,  AFSPC  and  the  Space  and  Missile  Systems 
Center  (SMC).  ERAM  will  be  initially  utilized  at  Aerospace  Corporation’s  Los  Angeles  based 
space  Concept  Design  Center  (CDC)  providing  decision-makers  insight  into  timeline  estimations 
and  probabilities  of  program  success  for  various  technical  concepts  based  on  historical 
comparisons.  ERAM  also  has  the  potential  to  be  used  as  a  training  tool  for  Air  Staff,  AFSPC  and 
SMC  personnel  to  better  understand  existing  organizational  interdependencies  and  required 
processes  necessary  to  acquire  a  capability  on  schedule  and  within  budget. 
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I. 


Introduction 


1.1  Background 

The  United  States  DoD’s  global  military  dominance  has  been  achieved  in  part  by  its 
ability  to  maintain  a  technological  advantage  over  its  adversaries.  As  research  and  development 
efforts  discover  new  technologies  and  potential  military  applications,  three  systems,  the  DoD 
JCIDS  system,  the  Program,  Planning,  Budget  and  Execution  (PPB&E)  system  and  the 
acquisition  system,  function  simultaneously  and  dependently  on  one  another  to  continue 
evolving  US  military  capabilities.  This  complex  “system  of  systems”  is  governed  by  numerous 
statutes,  policies,  instructions  and  guides  that  establish  a  process  framework  outlining  the 
activities  involved  and  how  they  relate  to  one  another.  It  requires  significant  experience, 
education  and  initiative  to  understand  and  successfully  navigate  the  system  as  a  requirements  or 
acquisition  professional.  DoD  weapon  systems  have  significantly  increased  in  complexity  and 
interdependency  in  the  past  two  decades  with  mandatory  Key  Performance  Parameters  (KPP) 
relating  to  net  centricity,  joint  interoperability,  as  well  as  several  others.  According  to  a 
Government  Accountability  Office  (GAO)  report,  programs  such  as  Transformational  Satellite 
Communications  System  (TSAT)  and  Space  Radar  were  “among  the  most  complex  programs 
ever  undertaken”  (Nelson  and  Sessions  2006).  DoD’s  ability  to  deliver  within  original  cost  and 
schedule  estimates  has  diminished.  In  a  GAO  survey  with  an  unspecified  total  number  of  MDAP 
PMs  interviewed,  “45  program  managers  (PM)  responded  that  their  program  had  been 
rebaselined  one  or  more  times  for  cost  and  schedule  increases”  (Nelson  and  Sessions  2006). 
Numerous  senior  DoD  leaders  have  indicated  that  one  of  the  reasons  for  this  is  due  to  insufficient 
and/or  inaccurate  data  used  to  make  early  decisions  to  pursue  material  solutions;  this  recognition 
has  placed  new  emphasis  on  improving  early  systems  engineering  analysis  during  capability  gap 
analysis  and  requirements  generation,  pre-Materiel  Development  Decision  (MDD)  activities,  and 


early  acquisition  activities  pre  MS-A.  President  Barack  Obama  released  the  National  Space 
Policy  of  the  United  States  of  America  on  June  28,  2010,  and  stated,  “Departments  and  agencies 
shall:  Improve  timely  acquisition  and  deployment  of  space  systems  through  enhancements  in 
estimating  costs,  technological  risk  and  maturity,  and  industrial  base  capabilities”  (Obama, 
National  Space  Policy  of  the  United  States  of  America  28  June  2010).  Secretary  of  Defense 
Gates  made  the  following  statement  in  his  Defense  Budget  Recommendation  Statement  on  6 
April,  2010, 

. . .  Second,  we  must  ensure  that  requirements  are  reasonable  and  technology  is 
adequately  mature  to  allow  the  department  to  successfully  execute  the  programs. .  .Third, 
realistically  estimate  program  costs,  provide  budget  stability  for  the  programs  we  initiate, 
adequately  staff  the  government  acquisition  team,  and  provide  disciplined  and  constant 
oversight  (SECDEF  2009). 

In  the  Chief  of  Staff  of  the  Air  Force’s  Vector  dated  4  July,  2010,  General  Norton 
Schwartz’s  fifth  priority  to  recapture  acquisition  excellence  states,  “Ultimately,  the  health  of  the 
Air  Force  requires  that  we  bring  acquisition  costs  and  timelines  under  much  greater  control  and 
oversight.”  In  an  effort  to  accomplish  this,  he  states  we  must  “Recapture  a  vision  for  aggressive 
science  and  technology  (S&T)  development,  the  rapid  transition  of  innovative  technology  into 
operational  capabilities,  and  harnessing  aerospace  technology  to  meet  broader  national  security 
needs”  (Schwartz  2010).  In  August,  2010,  Ft  Gen  Shackelford  (SAF/AQ)  was  the  keynote 
speaker  at  the  Air  Force  Systems  Engineering  conference  and  highlighted  two  specific 
challenges:  the  first  addressed  planning  and  “the  lack  of  technical  input  to  make  informed 
decisions”.  The  second  focused  on  execution  and  stated  that  “technical  issues/risks  aren’t 
discovered  and  addressed  at  the  right  time  and  at  the  right  level”  (Shanley  2011). 

The  President  of  the  United  States,  the  Secretary  of  Defense  and  senior  Air  Force 
officials  have  clearly  recognized  the  need  to  improve  the  capability  development,  requirements 
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validation  and  acquisitions  within  the  DoD  and  have  issued  strategic  guidance  to  tackle  the 
challenge  (Obama  28  June  2010,  Secretary  of  Defense  Gates  6  Apr  2009,  Office  of  the  Assistant 
Secretary  of  the  Air  Force  2009).  One  of  the  fundamental  factors  resulting  in  poor  program 
performance  is  insufficient  early  systems  engineering  analysis  and  inaccurate  Technology 
Readiness  Assessments  (TRA)  in  the  requirements  and  acquisition  systems.  This  has  inevitably 
resulted  in  early  program  decisions  being  made  based  upon  poor  cost  and  schedule  estimates 
(Shanley  2011). 

In  September,  2009,  a  doctoral  dissertation,  “Identifying  Enterprise  Leverage  Points  in 
Defense  Acquisition  Program  Performance”,  was  published  with  the  goal  of  characterizing  the 
system  of  acquiring  large,  complex,  socio-technological  systems  for  the  DoD  (Wirthlin  2009). 
The  research  resulted  in  an  in-depth  analysis  of  the  discrete  events  and  products  required  for  a 
typical  aerospace  defense  program  throughout  the  lifecycle,  with  emphasis  placed  on  events 
prior  to  MS-C.  Data  about  the  events  and  products  was  gathered  through  analysis  of  existing 
policy  and  guidance  (pre  2006)  as  well  as  numerous  interviews  with  defense  acquisition  experts. 
Subsequently,  this  data  was  modeled  and  programmed  using  the  Arena®  software  modeling  tool, 
resulting  in  the  first-ever  discrete-event  simulation  of  the  entire  defense  acquisition  model,  albeit 
abstracted  at  a  high  level.  For  the  purpose  of  this  research,  this  version  of  the  model  will  be 
referred  to  as  the  ERAM,  version  1.0  (ERAM  1.0).  ERAM  1.0  simulated  various  activities  and 
events  using  probabilities  for  decision  event  outcomes  and  timeline  distributions  to  determine 
likely  overall  program  timelines  as  well  as  probabilities  of  successful  program  execution  up  to 
MS-C.  The  output  is  reflected  by  a  probabilistic  determination  of  likely  program  duration 
through  approval  of  MS-C  using  Monte  Carlo  simulation  techniques.  ERAM  1.0  modeled  a 
capability  concept  or  idea  entering  the  Major  Command  (MAJCOM)  requirements  process  and 
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simulated  its  path  to  termination.  Additional  implementation  included  decision  points 
throughout  the  process  flow  to  include  determination  of  a  successful  design  review,  rework, 
funding  checks  and  other  events.  The  activities  for  discrete  events  incorporated  triangular 
distributions  for  elapsed  times.  For  example,  timelines  associated  with  affordability  assessments, 
preparing  for  reviews,  writing  documents,  all  had  probability  distributions  with  a  best  case,  worst 
case  and  most  likely  number  of  days,  with  the  data  elicited  from  various  sources.  These 
sequences  of  events  with  timeline  probabilities  and  decision  points  as  executed  through  the 
simulation  delivered  results  enabling  further  analysis  for  decision-making.  Accordingly,  with 
additional  refinement  and  enhancements  this  model  has  the  potential  to  be  used  as  a  valuable 
data  source  for  decision  makers  in  forecasting  a  program’s  development  and  delivery  schedules 
and  life  cycle  costs  (Wirthlin  2009). 

In  the  Fall  of  2010,  the  SMC,  Development  Planning  branch  (SMC/XR)  sponsored 
further  research  and  analysis  on  ERAM  1.0,  focusing  specifically  on  early  space  requirements 
and  acquisition  activities  pre  MS  B  (Figure  1).  The  model  will  be  updated  and  enhanced  to 
address  these  issues  specifically. 
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Figure  1:  Requirements  Generation  Process  (CJCS  2011) 

Research  data  was  collected  through  a  series  of  interviews  with  space  requirements  and 

acquisition  personnel  from  AFSPC  Requirements  directorate  (AFSPC/A5),  the  Under  Secretary 
of  the  Air  Force,  Acquisitions  Science,  Technology  and  Engineering  (SAF/AQR)  division  and 
space  acquisition  program  managers  coupled  with  an  analysis  of  updated  requirements  and 
acquisition  guidance.  The  research  approach  resulted  in  several  updated  model  versions,  ERAM 
1.1,  1.2,  2.0  and  2.1,  highlighted  and  explained  in  Figure  2  and  Table  1. 
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Figure  2:  ERAM  Evolution 


Table  1:  ERAM  Versions 


Model 

Version 

Description 

ERAM  1.0 

Original  model  developed  and  published  in  Wirthlin’s  dissertation 

ERAM  1.1 

Incorporated  updated  changes  from  Arena®  and  port  to  ExtendSim  software 

ERAM  1.2 

Incorporated  changes  from  DoDI  5000.02  May  2003  version  to  Dec  2008  version 

ERAM  2.0 

Incorporated  model  parameter  adjustments  to  capability  (i.e.  SPO  capability 
matrix) 

ERAM  2.1 

Incorporates  early  requirements  development  activities  from  JCIDS 

First,  ERAM  1 .0  was  ported  to  the  ExtendSim®  modeling  software  to  support  the  installed 
software  base  of  the  sponsor.  It  then  went  through  the  iterations  as  mentioned  in  Table  1 . 

The  main  research  focus  spans  ERAM  1 .2  through  ERAM  2. 1  by  making  ERAM  current 
with  recent  DoD  Instruction  5000.02  (DoDI)  updates  and  expanding  the  pre  MS-B  JCIDS 
processes  to  show  how  Development  Planning  (DP)  and  CCTD  processes  are  integrated  into  the 
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process  of  developing  a  new  capability.  The  research  found  capabilities  desired  by  the  AFSPC 
can  typically  be  categorized  as  either  “evolutionary”  or  “revolutionary”.  Evolutionary 
capabilities  are  achieved  through  modification(s)  or  upgrade(s)  to  an  existing  system(s).  In  these 
instances,  AFSPC  performs  its  JCIDS  requirements  process  and  may  reach  back  to  the  product 
center’s  system(s)  program  offices  (SPO)  for  their  DP.  On  the  other  hand,  revolutionary 
capabilities  are  typically  those  that  require  the  development  of  an  entirely  new  system.  For  these 
DP  activities,  AFSPC  reaches  out  to  subject  matter  experts  (SME)  at  SMC/XR.  These 
revolutionary  capabilities  have  the  potential  of  moving  out  of  SMC/XR  and  into  a  new  SPO. 
Additionally,  the  research  made  inquiries  into  deviations  from  existing  policy  and  methods  of 
approval  for  these  deviations.  This  served  as  a  way  to  enrich  the  realism  of  the  model’s  and 
subsequently,  its  usefulness  for  decision  makers. 

1.2  Problem  Statement 

President  Obama  states  in  the  National  Security  Strategy  for  the  United  States  of 
America, 

Cost-effective  and  efficient  processes  are  particularly  important  for  the  DoD,  which 
accounts  for  approximately  70  percent  of  all  Federal  procurement  spending.  We  will 
scrutinize  our  programs  and  terminate  or  restructure  those  that  are  outdated,  duplicative, 
ineffective,  or  wasteful.  The  result  will  be  more  relevant,  capable,  and  effective  programs 
and  systems  that  our  military  wants  and  needs  (Obama,  National  Security  Strategy  2010). 

Furthermore,  from  the  National  Defense  Strategy  (June  2008),  Secretary  of  Defense  Robert 

Gates  states,  “We  also  must  continue  to  improve  our  acquisition  and  contracting  regulations, 

procedures,  and  oversight  to  ensure  agile  and  timely  procurement  of  critical  equipment  and 

materials  for  our  forces”  (Gates,  National  Defense  Strategy  2008). 

Senior  leader  decisions  with  regards  to  future  space  programs  must  be  based  upon 

realistic  and  accurate  cost  and  schedule  estimates  which  historically  have  not  existed.  Sadly, 

cost  overruns  and  schedule  delays  have  become  the  norm  in  space  acquisitions.  National  defense 
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leadership  has  emphasized  the  importance  of  continuing  to  expand  our  role  in  space  as  well  as 
increasing  space  capabilities  in  a  costly  and  timely  manner.  Currently,  there  is  no  existing 
capability  or  process  to  quickly  and  comprehensively  develop  the  requirements  and  acquisition 
program  details  for  large  and  complex  space  systems.  According  to  Dr.  Peter  Hantos  and  Nancy 
Kern  of  the  Aerospace  Corporation,  “Pre  MS-A  Systems  Engineering  and  Pre  MS-B  Software 
Engineering  efforts  are  not  comprehended  in  any  estimation  models”  (Hantos  and  Kern  2011). 

1.3  Implications 

The  primary  purpose  of  this  research  was  to  update  and  modify  ERAM  1.0  to  ERAM  2.1 
for  SMC/XR  to  utilize  as  a  tool  to  increase  the  fidelity  of  CCTDs.  By  improving  the  quality  and 
viability  of  the  data  within  these  documents,  Defense  Acquisition  Board  (DAB)  decisions 
regarding  future  space  capabilities  will  be  based  on  more  accurate  cost  and  schedule  estimates 
resulting  in  decreased  breaches  and  improved  program  performance. 

Additionally,  there  are  several  other  important  implications  that  modeling  the 
requirements  and  acquisition  process  delivers.  ERAM  in  conjunction  with  this  comprehensive 
report  can  aid  requirements  and  acquisition  professionals  to  better  understand  organizational 
relationships  and  provide  a  map  of  required  activities  based  upon  current  directives  and 
guidance,  improving  the  likelihood  of  successfully  maintaining  program  cost,  schedule  and 
performance  objectives.  The  model  could  also  help  OSD  and  SAF  personnel  identify 
inefficiencies  and  disconnects  within  the  vast  number  of  current  instructions,  resulting  in 
modification  and  simplification  of  current  guidance.  ERAM  also  provides  detailed  context  to  the 
well  known,  but  too  often  incomprehensible,  DAU  acquisition  “wall  chart.”  This  chart  provides 
a  high  level  picture  of  acquisition  activities,  but,  lacks  detail  to  effectively  train  acquisition 
professionals.  By  supplementing  the  “wall  chart”  with  the  ERAM  framework,  real  world  data 
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can  significantly  improve  the  current  training  curriculum  for  acquisition  personnel —  possibly 
resulting  in  a  “flight  simulator”  of  the  acquisition  system.  This  training  device  could  evolve  into 
a  more  disciplined  and  rigorous  training  &  evaluation  system  for  acquisition  personnel. 

Chapter  II  (Literature  Review)  of  this  report  will  provide  an  overview  of  the  key 
documents  that  heavily  influenced  the  development  of  ERAM  1.2  through  2.1  and  the  findings 
included  in  this  report.  Chapter  III  (Methodology)  describes  the  methods  used  to  collect  the  data 
necessary  to  accurately  update  ERAM.  Furthermore,  it  highlights  the  methodology  for  the 
research  accomplished  through  interviews  of  subject  matter  experts  and  processes  for  making 
updates  to  the  software  model.  Chapter  IV  (Results  and  Analysis)  will  consolidate  the  findings 
from  the  interviews  and  discuss  how  they  were  implemented  into  the  model.  Chapter  V 
(Conclusions  and  Recommendations)  summarizes  the  findings  of  this  research  effort  and 
provides  recommendations  on  additional  research  and  policy  changes  that  would  benefit  space 
acquisitions. 
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II. 


Literature  Review 


Research  on  the  subject  of  DoD  capability  development  required  the  review  of  more  than 
50  policy  documents,  official  instructions,  guidance,  journal  articles,  and  briefings.  Table  2 
highlights  the  eight  major  resources  that  are  the  seminal  pieces  of  literature  for  this  research. 
They  were  utilized  to  capture  the  essence  of  the  space  acquisition  and  requirements  themes  for 
making  modeling  decisions  and  acquisition  observations  for  this  research  effort.  However,  these 
are  but  a  small  snapshot  of  the  policy  requirements  that  acquisition  professionals  will  need  to 
quickly  process  and  internalize  in  order  to  develop  requirements  and  acquisition  program  details 
for  complex  space  systems.  Wirthlin’s  dissertation  (Resource  #1  below)  provides  the  foundation 
of  this  modeling  effort.  The  remainder  provides  significant  guidance  in  the  ever-changing  world 
of  acquisition.  Each  of  these  resources  is  intended  to  aid  in  the  background  required  for  inputs 
into  ERAM  development.  The  foundation  of  Wirthlin’s  dissertation  and  the  additional  guidance 
provided  in  the  literature  review  serve  as  the  basis  for  the  information  which  can  be  utilized  for 
improved  decision-making  and  closing  the  gap  in  acquisition  knowledge.  More  details  about 
each  resource  follow. 
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Table  2:  Highlighted  Literature  Overview 


Resource 

1) 

Wirthlin,  J.  Robert.  Identifying  Enterprise  Leverage  Points  in  Defense  Acquisition 
Program  Performance. 

2) 

Gates,  Robert  M.  SECDEF  MEMO.  Department  of  Defense  Efficiency  Initiatives. 

16  Aug  2010. 

3) 

Carter,  Ashton  B.  USD/AT&L  MEMO  for  Acquisition  Professionals:  Better  Buying 
Power:  Guidance  for  Obtaining  Greater  Efficiency  and  Productivity  in  Defense 
Spending.  14  Sep  2010. 

4) 

Assistant  Secretary  of  the  Air  Force  for  Acquisition  (SAF/AQ)  Early  Systems 
Engineering  Guidebook  Version  1. 

5) 

Loren,  Jeff  C  (SAF/AQRE).  The  ABCs  of  Concept  Evolution:  A  Better-Informed 
Materiel  Development  Decision  for  USAF  Programs.  1 8  Nov  2010 

6) 

United  States  of  America  Congress.  Weapons  Systems  Acquisition  Reform  Act 
(WSARA)  of 2009 

7) 

Acquisition  Improvement  Plan  (AIP)  4  May  2009 

8) 

Directive-Type  Memorandum  (DTM)  09-025  -  Space  Systems  Acquisition  Policy 
(SSAP) 

Resource  #1:  Identifying  Enterprise  Leverage  Points  in  Defense  Acquisition  Program 
Performance 

Wirthlin’s  PhD  dissertation  simulated  the  acquisition  “wall  chart”  using  the  Arena® 
software  tool.  It  provides  a  thorough  background  of  the  acquisition  process,  the  PPBE  process, 
and  the  JCIDs  process.  Through  reviewing  literature  and  interviewing  members  of  the  DoD 
requirements  and  acquisition  communities,  this  dissertation  provides  a  candid  view  of  many  of 
the  challenges  and  opportunities  to  bring  an  acquisition  program  to  MS-C. 

The  fundamental  research  questions  in  the  dissertation  were  the  following:  “How  does 
the  acquisition  system  work?”  A  follow  up  question  was,  “Why  does  the  system  behave  the  way 
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that  it  does?”  And  finally,  “Are  there  things  that  can  be  done  to  improve  the  system?”  (Wirthlin 
2009)  Throughout  the  dissertation  and  model  development,  real  world  detailed  examples  were 
given  on  not  only  how  the  acquisition  process  should  work  according  to  policy  and  guidance,  but 
also  how  MAJCOMs,  Air  Staff,  or  SPO  personnel  personalized  the  system  to  be  more  effective 
within  their  own  areas  of  responsibility.  ERAM  1.0,  as  developed  in  the  Arena®  simulation 
program,  captured  process  steps,  activities,  and  injected  uncertainty  to  show  additional  realism. 
Although  it  captured  various  elements  as  discrete  events,  these  were  modeled  with  a  triangular 
distribution  of  the  number  of  days  required  to  complete.  Inputs  to  these  discrete  event  activities 
were  minimum,  maximum  and  most  likely  number  of  days  based  on  a  user’s  actual  experience 
and  estimates.  Alternative  paths  through  the  system  or  branching  points  were  also  identified 
with  associated  probabilities.  For  example,  one  branching  point  was  during  a  source  selection. 
The  probability  of  receiving  a  protest  from  a  losing  contractor  was  identified  as  20%.  ERAM  1 .0 
captured  this  probability  and  added  an  activity  to  account  for  the  protest  event. 

ERAM  1 .0  represents  a  predictive  model  of  various  detailed  elements  of  the  acquisition 
processes  with  their  timelines  and  interdependencies  along  with  several  exploratory  variants  that 
include:  Air  Staff  intervention;  MAJCOM  approval  body(s)  intervention;  technical  interventions; 
interventions  at  different  design  reviews;  funding  stability;  etc.  By  exploring  these  variants,  and 
combining  these  adjustments  over  acquisition  category  (ACAT)  I,  II,  and  III  level  programs, 
conclusions  were  drawn  regarding  ways  to  improve  program  outcomes.  An  identical  analysis 
approach  will  also  be  used  in  this  new  research  by  focusing  on  the  SMC/XR  processes  in  the 
development  planning  phase  of  acquisition. 

The  “meat”  of  the  dissertation  was  certainly  ERAM  1 .0,  its  development,  its  results,  and 
the  verification  and  validation.  The  printed  version  of  ERAM  1.0  in  a  readable  format  takes  a 
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roll  of  butcher  block  paper  approximately  14  feet  long.  The  main  outputs  include:  where  the 
process  was  terminated;  the  probability  of  reaching  the  exit  points,  and  the  number  of  days 
required  to  complete  the  process  through  MS-C  and  other  areas.  A  concept  or  idea  enters  the 
model  at  the  requirements  shop.  The  exact  method  for  a  concept  entering  the  requirements 
process  is  not  specifically  modeled  in  ERAM  1 .0.  It  assumes  an  idea  is  introduced  to  the 
requirements  organization  and  then  proceeds  forward.  This  is  then  carried  forward  and  either 
terminates  at  various  points  along  ERAM  1.0,  or  reaches  MS-C.  Innovative  methods  to  account 
for  the  various  unplanned  taskings  levied  on  a  program  were  modeled  using  uncertainty  events  at 
a  regular  frequency. 

Uncertainty  driven  by  political  circumstances  is  artificially  modeled  by  randomly 
generating  a  ‘program  review’  where  the  finances,  program  management  and  other 
aspects  of  a  program  are  ‘reviewed’  for  potential  cuts  and/or  changes.  A  set  driver  of 
uncertainty,  also  artificially  driven,  is  named  simply  ‘event  happens’  and  is  used  to 
account  for  the  stochastic  nature  of  problems  encountered  in  the  execution  of  the 
development  program,  running  the  gamut  from  the  impacts  of  ‘known  unknowns’  to 
“unknown  unknowns”  (Wirthlin  2009). 

Examples  of  these  uncertain  events  could  include  some  tasking  from  Air  Staff  to  write  a  point 
paper  defending  their  acquisition  strategy.  This  is  an  out-of-cycle  activity  levied  on  the  PM, 
which  may  or  may  not  happen.  So  therefore,  each  “uncertainty  event”  captured  the  possibility  of 
an  unplanned  activity  path. 

The  dissertation  and  ERAM  1 .0  lays  the  groundwork  for  observers  to  take  a  hard  look  at 
how  policy  changes  impact  the  acquisition  process.  The  insight  gained  from  inserting  realism 
into  the  acquisition  “wall  chart”  or  “horse  blanket”  can  be  enlightening  to  those  wishing  to 
understand  the  impact  of  certain  decisions  on  the  overall  outcome  of  a  program.  The  analogy  of 
the  “butterfly  effect”  is  that  a  butterfly  flapping  its  wings  at  one  location  could  potentially  create 
a  major  windstorm  at  another  location.  A  small  tweak  to  the  acquisition  process  could  have  a 
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major  impact  to  the  outcome.  It  has  the  potential  to  aid  decision  makers  to  understand  how  to 
resource  their  programs,  provide  more  realistic  expectations  on  program  timelines,  and  in  the 
future,  provide  “knobs”  where  critical  process/program  parameters  could  be  modified  to  improve 
overall  outcomes  (sensitivity  analysis).  A  high-level  view  of  this  model  is  found  in  Figure  3. 
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Figure  3:  ERAM  1.0  Arena  (Wirthlin  2009) 


Resource  #2:  Department  of  Defense  Efficiency  Initiatives 

Secretary  of  Defense  Gates  provided  direction  for  eliminating  excess  costs  in  the  DoD 
budget  in  a  memo  issued  16  August  2010.  These  efficiencies  are  intended  to  align  the  Defense 
budget  with  the  overall  intent  to  reduce  the  size  of  the  federal  budget.  Some  of  these 
efficiencies  include:  “10%  reduction  in  funding  for  service  support  contractors”;  freezing  the 
number  of  staff  billets  at  several  agencies  including  Office  of  the  Secretary  of  Defense  (OSD); 
25%  reduction  in  advisory  studies;  eliminate  Assistant  Secretary  of  Defense  for  Networks  & 
Information  Integration  (ASD/NII)  and  J6  organizations;  and  “complete  a  comprehensive  review 
of  all  Department-required  oversight  reports  with  the  aim  of  reducing  the  volume  by  eliminating 
non-essential  requirements”  (OUSD/ATL  2010).  This  impacts  the  acquisition  process  in  several 
ways  such  as  extending  activity  timelines  with  a  less  experienced  and  minimally  staffed 
workforce.  Additionally,  future  iterations  of  the  model  could  have  activities  eliminated  if  they 
are  considered  to  be  surplus  oversight  requirements.  These  efficiencies  have  the  potential  to 
provide  benefit  in  reduced  bureaucracy  and  an  improvement  in  the  ability  of  acquisition 
professionals  to  understand  the  requirements  levied  on  them.  All  of  these  must  be  captured  in 
ERAM. 

Resource  #3:  Better  Buying  Power:  Guidance  for  Obtaining  Greater  Efficiency  and  Productivity 
in  Defense  Spending 

This  letter  also  provides  several  key  directives  to  redirect  $100B  in  defense  spending  by 
improving  business  practices  and  cutting  lower  priority  spending  initiatives.  Secretary  Carter 
stated:  “To  put  it  bluntly:  we  have  a  continuing  responsibility  to  procure  the  critical  goods  and 
services  our  forces  need  in  the  years  ahead,  but  we  will  not  have  ever-increasing  budgets  to  pay 
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for  them.  We  must  therefore  strive  to  achieve  what  economists  call  productivity  growth:  in 

simple  terms,  to  DO  MORE  WITHOUT  MORE.”  (Carter  2010)  Some  key  elements  include 

the  following:  1)  At  MS-A,  the  acquisition  decision  memorandum  (ADM)  will  include 

affordability  as  a  Key  Performance  Parameter  (KPP);  2)  Eliminate  redundancies  within 

portfolios;  3)  Competitive  Strategy  required  at  all  milestones;  4)  Further  insight  into  contractors 

Independent  Research  and  Development  (IRAD)  investments;  5)  Modify  Nunn-McCurdy  rules 

for  special  situations;  etc.  The  independent  cost  estimates  (ICE)  will  now  be  used  to  drive 

productivity  into  the  programs  and  provide  incentives.  There  will  now  be  “should  cost”  and 

“will  cost”  estimates.  Previously,  only  the  “will  cost”  estimates  were  used.  The  “should  cost” 

estimates  will  now  be  inserted  into  the  process  to  incentivize  leanness  and  efficiencies  by  the 

PM.  If  prices  are  negotiated  less  than  the  “will  cost”  and  the  program  is  executed  as  such,  cost 

savings  could  be  reallocated  within  programs  to  acquire  other  capabilities. 

“The  metric  of  success  for  Should  Cost  management  leading  to  annual  productivity 
increases  is  annual  savings  of  a  few  percent  from  all  our  ongoing  contracted  activities  as 
they  execute  to  a  lower  figure  than  budgeted.  Industry  can  succeed  in  this  environment 
because  we  will  tie  better  performance  to  higher  profit,  and  because  affordable  programs 
will  not  face  cancellation.”  (Carter  2010) 

Cost  estimates  through  OSD  will  impact  program  flows  and  timelines.  These  cost  drivers  impact 
program  timelines  in  the  modeling  and  analysis  as  the  OSD  governing  body  for  cost  estimates 
adds  additional  checks  and  balance  in  the  early  systems  engineering  process.  A  fair  criticism  of 
space  programs  would  be  that  the  “will  cost”  estimates  were  not  achieved  suggesting  the  notion 
of  “should  cost”  estimates  as  being  without  merit.  However,  with  the  added  emphasis  on  cost 
estimation  with  more  defined  processes,  the  “should  cost”  will  be  a  driving  force  in  future 
acquisitions  programs.  Further  research  will  determine  if  this  new  emphasis  achieved  its  desired 
goals.  As  acquisition  professionals  are  given  the  knowledge  tools  to  execute  their  programs, 
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they  will  be  empowered  to  successfully  navigate  through  the  information  and  succeed.  These 
must  be  accounted  for  and  captured  in  ERAM. 

Resource  #4:  Early  Systems  Engineering  Guidebook 

This  guidebook  is  critical  for  this  research  project  as  it  discusses  the  “meat”  of  the 
acquisition  process  prior  to  MS-A.  It  discusses  the  Capability  Based  Assessment  (CBA), 
Concept  Exploration  and  Refinement,  AoA,  the  CCTD  document,  and  the  Materiel  Solution 
Analysis  phase. 

“A  development  and  acquisition  organization,  typically  XR,  responsible  for  translating 
high-level  system  needs  into  more  detailed  system-level  information.  With  the  help  of  all 
stakeholders,  they  generate  and  analyze  alternative  system  concepts,  and  provide 
balanced  estimates  of  effectiveness,  performance,  cost,  schedule,  and  risk  to  assist  the 
stakeholders  in  selecting  preferred  concepts”  (SAF/AQ  2009). 


Figure  4:  Early  Systems  Engineering  Process  (SAF/AQ  2009) 
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Figure  4  highlights  the  system  engineering  “V”  with  the  associated  organizational  impacts 
throughout  the  process.  Thorough  early  systems  engineering  efforts  have  the  potential  to  close 
knowledge  gaps.  These  CCTD  processes  are  recommended  as  updates  to  ERAM  in  Version  2.1. 
Resource  #5:  The  ABCs  of  Concept  Evolution:  A  Better-Informed  Materiel  Development 
Decision  for  USAF  Programs 

The  Concept  Initiation  briefing  addresses  the  MDD  leading  up  to  MS-A.  It  addresses  the 
relationships  between  systems  engineering,  DP,  S&T,  and  the  CCTD.  It  acknowledges  changes 
to  the  JCIDS  that  eliminated  the  Functional  Solutions  Analysis  (FSA)  and  the  Analysis  of 
Material  Alternatives  (AMA)  but  the  knowledge  resulting  from  these  types  of  studies  is  still 
required  to  support  MDDs  and  MS-As.  It  discusses  a  paradigm  shift  where  the  shift  is  from 
technology  focus  to  capability  focus.  Early  systems  engineering  is  critical  to  transform  the 
technology  into  meeting  a  capability  gap.  CCTDs  are  becoming  a  critical  element  of  the  early 
systems  engineering  process  to  support  MDDs  and  AoAs.  (Loren  2010)  Early  systems 
engineering  as  updated  in  the  model  has  the  potential  to  impact  later  impacts  to  the  process. 
Understanding  these  early  system  engineering  impacts  may  reduce  overall  program  cost  as 
decision  makers  have  the  necessary  knowledge.  These  references  and  activities  need  to  be 
incorporated  into  ERAM. 

Resource  #6:  Weapons  Systems  Acquisition  Reform  Act 

WSARA  updates  Title  10  of  the  US  Code  in  several  areas  to  improve  the  acquisition 
process  in  DoD.  For  example,  to  improve  the  cost  estimates  for  major  defense  acquisition 
programs  (MDAP),  a  Director  of  Cost  Assessment  and  Program  Evaluation  (CAPE)  is  directed 
to  place  an  improved  emphasis  on  cost  estimation.  Other  events  impacting  the  model  include 
competitive  prototyping  requirements  which  impact  acquisition  and  contracting  strategies.  This 
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is  related  to  the  new  adjustments  to  DoDI  5000.02  policy  (US  Congress  2009).  Additional 
oversight  has  the  potential  to  provide  more  accurate  cost  estimates  with  less  of  an  impact  for  cost 
growth  and  program  breaches.  Although  these  cost  estimates  may  add  time  early  in  the  process, 
the  PMs  may  have  a  more  accurate  assessment  of  their  program  This  would  potentially  reduce 
later  program  inquiries  from  Congress  or  Headquarters  personnel.  More  accurate  program  cost 
estimates  aid  in  closing  knowledge  gaps  for  acquisition  professionals.  ERAM  needs  to  explicitly 
model  these  changes  and  impacts. 

Resource  #7:  Acquisition  Improvement  Plan 

As  a  follow-up  to  WSARA,  the  Air  Force  made  a  commitment  to  improve  various 
acquisition  processes  based  on  lessons  learned  from  past  acquisition  programs  with  negative 
outcomes.  Secretary  Donley  and  General  Schwartz  stated:  “This  plan  focuses  our  efforts  and 
will  serve  as  our  strategic  framework  for  the  critical  work  of  modernizing  and  recapitalizing  our 
air,  space  and  cyber  systems”  (SAF/AQ  2009).  One  of  the  goals  is  to  improve  the  core 
workforce  in  the  acquisition  community.  “To  operate  effectively,  today’s  acquisition  workforce 
must  be  supported  by  a  human  resource  environment  that  recognizes  the  complexity  of  the 
acquisition  mission  and  grooms  professional  journeyman  as  well  as  future  leaders  in  all  of  the 
acquisition  functional  specialties”  (SAF/AQ  2009).  This  is  especially  true  as  this  research 
reports  on  observations  by  key  personnel  in  the  acquisition  community  on  the  requirements  for 
knowledgeable  team  members.  Additionally,  this  document  identifies  the  need  to  improve  the 
requirements  generation  process.  “In  the  future,  there  will  be  acquisition  involvement  earlier  in 
the  Air  Force  requirements  and  development  process  and  systems  engineering  techniques  will  be 
applied  to  assist  in  the  tradeoffs  that  occur  as  part  of  the  process”  (SAF/AQ  2009).  The  key  to 
this  is  in  ensuring  the  constant  collaboration  with  Headquarters,  lead  commands,  and  acquisition 
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program  offices.  These  interpersonal  relationships  and  collaboration  will  aid  PMs  to  internalize 

all  of  the  required  information.  Such  changes  will  need  to  be  accommodated  in  ERAM. 

Resource  #8:  Space  Systems  Acquisition  Policy 

Ashton  Carter,  USD/AT&L,  directed  specific  acquisition  procedures  for  space  programs. 

Some  of  these  activities  go  above  and  beyond  the  standard  DoDI  5000.02  guidance  and  impact 

the  modeling  development  efforts  in  this  research. 

“The  Milestone  Decision  Authority  (MDA)  shall  conduct  a  formal  program  assessment 
following  the  System  Design  Review  (SDR)  for  space  systems.  The  SDR  provides  an 
opportunity  to  assess  satisfaction  of  user  needs  through  functional  decomposition  and 
traceability  of  requirements  from  the  initial  capabilities  document  (ICD)  to  the  contractor’s 
functional  baseline  and  system  specification.  An  Independent  Program  Assessment  (IP A) 
shall  be  provided  to  support  the  Pre-System  Design  Review  Assessment  (P-SDRA).”  (Carter 
2010) 

These  post  review  assessments  with  the  MDA,  to  include  optional  IPA  support,  add 
activities  with  timeline  distributions  to  the  model.  More  importantly  they  impact  the  actual 
execution  requirements  levied  on  a  SPO.  This  information  is  significant  to  this  research  as  it 
modifies  the  DoDI  5000.02  guidelines  specific  to  space  programs.  PMs  are  required  to  be 
prepared  for  these  multiple  MDA  looks  which  are  above  and  beyond  standard  MDA  reviews. 
Summary 

This  literature  review  demonstrates  the  ever-changing  pockets  of  guidance  and 
knowledge  levied  on  acquisition  professionals  within  DoD.  All  of  the  policy  changes  are 
intended  to  improve  program  execution.  However,  the  number  of  policies  and  their  frequency  of 
change  levy  vast  knowledge  and  training  requirements.  Wirthlin’s  dissertation  showed  the 
complexity  of  the  knowledge  network  PMs  are  required  to  distill  into  something  executable. 
Several  policy  documents  and  memos  in  this  review  showed  the  many  nuances  in  oversight 
required  at  various  stages  in  the  process.  Others  attempt  efficiency  improvements  in  acquisition. 
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Early  systems  engineering  was  identified  as  a  critical  element  in  program  improvement. 
Ultimately,  all  of  these  elements  show  areas  where  acquisition  professionals  need  to  quickly 
understand  requirements  and  acquisition  program  details  for  complex  space  programs. 
Incorporating  and  addressing  these  elements  in  ERAM  fills  an  important  knowledge  gap  for  both 
the  researcher  and  practitioner  of  weapon  system  acquisition. 
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III.  Methodology 


The  research  team  performed  qualitative  social  science  research  with  the  objective  of 
building  and  refining  an  existing  quantitative  discrete  event  simulation  model  to  be  utilized  by 
SMC  and  SPO  XR  branches  as  a  decision  support  tool  during  early  concept  analysis. 

3. 1  Research  Scope 

The  focus  of  this  research  is  to  analyze  and  model  the  discrete  events  for  Air  Force  space 
programs  from  capability  gap  analysis  through  MS-C  of  the  acquisition  system,  more 
specifically,  the  early  front-end  area  of  capability  gap  analysis,  requirements  validation,  DP  and 
early  acquisition.  Focusing  on  the  front  end  of  ERAM  1.0  limited  the  literature  reviews  and  key 
personnel  interviews  to  OSD,  Acquisition,  Technology  &  Logistics  (OSD/AT&L),  Secretary  of 
the  Air  Force,  Acquisitions  (SAF/AQ),  AFSPC,  SMC,  the  Aerospace  Corporation  and 
government  support  contractors.  Although  some  of  the  discussion  with  key  requirements 
personnel  included  the  cyber  and  information  technology  (IT)  domain,  the  modeling 
implementation  remained  focused  on  space  capability  requirements  and  programs. 

The  PPB&E  and  Test  and  Evaluation  (T&E)  processes  were  not  expanded  beyond  what 
was  previously  developed  in  ERAM  1.0.  PPB&E  is  the  calendar  driven  funding  process;  the 
detailed  formal  budgeting  process  will  be  a  topic  left  to  future  research  to  increase  the  fidelity  of 
ERAM.  Certain  elements  in  the  model  have  incorporated  checks  for  available  funding,  ICE,  and 
above-threshold  cost  increases,  however,  do  not  identify  the  specific  activities  and  decision 
points  as  well  as  their  duration  distributions  and  probabilities,  respectively.  Uncertainty  events 
which  occur  in  the  model  may  have  ties  to  budget  directives  and  decisions  and  will  be  based  on 
out-of-cycle  budget  cut  drills  and  other  events  gleaned  from  ERAM  1 .0 

Similarly,  analysis  of  the  T&E  activities  identified  in  ERAM  1.0  was  not  included  in  the 
scope  of  this  research.  Since  it  is  such  a  critical  and  timely  activity  in  any  capability 
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development,  this  is  a  prime  topic  for  additional  research,  e.g.  identify,  document  and  update 
ERAM  with  the  differences  between  aerospace  and  space  T&E  activities. 

3.2  Research  Objectives 

At  the  beginning  of  the  research  effort,  the  following  objectives  were  identified  and  agreed 
upon  by  both  the  customer  (SMC/XR)  as  well  as  the  research  team  (AFIT/ENV). 

1)  Review  and  update  ERAM  1.0  (predominantly  focused  on  post  MS-B  activities  and 
generalized  USAF  acquisition  processes)  to  ERAM  2.1  for  space  programs.  These  include: 

a.  Implement  corrections  identified  when  transferring  the  model  from  Arena®  to 
ExtendSim  modeling  software  (ERAM  1.1) 

b.  Ensure  updated  DoDI  5000.02  representation  and  acquisition  system  processes 
(ERAM  1.2) 

c.  Incorporate  modeling  parameter  knobs  to  adjust  certain  model  sensitivities,  e.g.  a 
SPO  Experience  matrix,  ACAT,  and  technology  readiness  levels  (TRL)  (ERAM  2.0) 

d.  Research  and  reflect,  where  applicable,  capability  gap  analysis,  JCIDS,  and  DP 
processes  (ERAM  2.1) 

2)  Validate  utility  of  the  model  and  how  it  can  be  used  in  existing  early  requirements  and 
acquisition  processes 

3)  Identify  additional  research  requirements  for  future  versions  of  ERAM 
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3.3  Methodology 

In  order  to  achieve  the  objectives  of  this  research,  a  five  phase  approach  was  established 
and  is  illustrated  in  Figure  5. 


PHASE  1: 

Information 
collection  &  review 

Review  policy,  memorandums,  instructions,  guides,  journal/research  pubs 
relating  to  capability  development,  requirements  validation  &  acquisitions 

PHASE  2: 

_ 

Familiarize  with  ERAM  1.0  in  Arena® 

Review  ERAM  1.0  in  ExtendSim®  (model  by  Aerospace  Corp.) 

PHASE  3:  \ 

Design  ERAM  1.1, 1.2,  2.0  and  2.1  by  applying 
information  from  Phase  1  to  ExtendSim®  ERAM  1.0 

PHASE  4: 

- — 

Implement  design  changes  in 

ExtendSim®  to  develop  new  ERAM 
versions 

PHASE  5: 

~~ 

Write  final  report  of 
findings  and  brief 

Figure  5:  Group  Research  Project  (GRP)  Phased  Approach 


3.3.1  Phase  1 

The  first  step  was  to  review  statutes,  official  DoD  and  Air  Force  regulations,  instructions, 
guides  and  publications  as  well  as  applicable  journal  articles  and  research  papers.  Figures  6  and 
7  provide  a  glimpse  into  the  complexity  and  interdependencies  of  the  policies  and  instructions. 
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Figure  6:  Ops/Ops  Support/T&E  (SMC/SLX  2011) 
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Figure  7:  Acq/Eng/Sustainment  Mngt/Quality  (SMC/SLX  2011) 

There  are  a  significant  number  of  references  that  establish  the  framework  for  how  to  develop 
and  deliver  new  space  capabilities.  Figure  8  represents  the  primary  sources  of  policy 
information  utilized  in  this  research  effort,  a  subset  of  instructions  from  Figures  6  and  7,  and 
what  portion  of  the  “system”  they  are  most  closely  related  to  (vertical  alignment).  The  challenge 
was  identify  the  linkages  between  the  activities  and  the  various  decision  points  that  occur 
between  capability  gap  analysis,  requirement  validation,  DP,  S&T  activities  and  executing  the 
acquisition. 
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Figure  8:  Pedigree  of  Guidance  for  Space  Capability  Development 

The  second  step  of  phase  one  was  to  conduct  interviews  with  SMEs  familiar  with  capability 

gap  analysis,  requirements  development  and  the  acquisition  process.  These  were  semi-structured 
interviews  using  purposeful  and  snowball  sampling.  Purposeful  sampling  identified  known 
SMEs  (military,  government  civilians,  Federally  Funded  Research  &  Development  Contract 
personnel  and  support  contractors)  from  SMC,  AFSPC  and  SAF/AQ,  each  of  which  had 
significant  experience  in  the  acquisition  and  requirements  career  fields.  The  interviews  provided 
a  perspective  from  where  the  policy  makers  and  senior  leaders  are  most  likely  to  impact 
timelines  and  coordination  cycles  on  required  acquisition  documentation.  “In  sociology  and 
statistics  research,  snowball  sampling  is  a  technique  for  developing  a  research  sample  where 
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existing  study  subjects  recruit  future  subjects  from  among  their  acquaintances”  (Castillo  2009). 
Through  the  course  of  the  interviews,  additional  names  of  SMEs  were  provided  that  could  help 
answer  specific  questions  regarding  activities  in  several  of  the  phases  of  the  capabilities 
development  system.  Some  of  the  questions  asked  include  the  following: 

1)  Describe  what  processes  you’ve  been  involved  with  regards  to  space  capability 
development  (i.e.  gap  analysis,  S&T,  JCIDS  documents,  DP,  and/or  acquisitions)  and 
with  which  program 

2)  Describe  the  specific  activities  and  decision  points  for  the  processes  you’ve  been 
involved  in  from  question  1 . 

3)  What  reviews  and  documentation  were  required  for  your  program? 

The  complete  list  of  representative  questions  can  be  found  in  Appendix  C. 

3.3.2  Phase  2 

After  completing  phase  one,  the  next  step  was  to  verify  the  current  baseline  program,  ERAM 
1.0,  had  been  transferred  correctly  from  Arena®  to  ExtendSim®  software.  Feedback  was 
provided  to  SMC/XR’s  Aerospace  software  engineers  with  any  discrepancies  or  recommended 
updates  identified.  The  primary  reason  for  transferring  the  system  to  ExtendSim®  was  to  lower 
overall  distribution  costs;  licensing  for  a  wider  distribution  of  Arena®  versions  of  the  modeling 
software  would  be  cost  prohibitive. 

In  addition  to  familiarizing  themselves  with  ERAM  1.0  in  ExtendSim®,  the  research  team 

reviewed  the  associated  doctoral  dissertation,  “Identifying  Leverage  Points  in  Defense 

Acquisition  Program  Performance”  and  met  with  the  author  on  a  regular  basis  to  better 

comprehend  the  methodology  used  to  create  the  ERAM  1.0  framework.  The  research  team  also 

participated  in  frequent  meetings  with  the  SMC/XR  Aerospace  team  that  implemented  the 

ExtendSim®  version  of  the  model.  A  hardcopy  of  the  ERAM  1.0  was  printed  out  to  help 

facilitate  visualization  of  the  process;  it  requires  9  large  butcher  paper  sheets  pasted  together  to 
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form  a  single  picture,  as  shown  in  the  photograph  in  Figure  9  (the  ninth  page  is  behind  Major 
Leach). 


Figure  9:  ERAM  1.0  Printout 

In  meetings  with  the  Aerospace  implementation  team  in  Los  Angeles,  the  printout  was 
displayed  in  several  locations  to  clarify  portions  of  the  model  and  identify  where  the  various 
corrections  and  updates  would  occur.  These  updates  would  include  developing  the  model 
triggers  identified  as  an  output  of  the  AFSPC  Integrated  Planning  Process  (IPP),  the  JCIDS 
document  development  activities  and  DoDI  5000.02  activities.  Figure  10  is  an  example  of  one 
portion  of  ERAM  close  up  to  see  individual  elements. 
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Program  Kill  at  MDA after  SDR 


Figure  10:  ERAM  1.0  Close  Up  View 


For  purposes  of  this  discussion,  five  of  the  many  ExtendSim®  icons  are  described  and 
illustrated  in  Figure  11.  The  “Event/ Activity”  icon  is  implemented  with  a  time  duration  allowing 
a  distribution  to  be  selected.  For  this  research,  triangular  distributions  were  implemented  based 
on  the  data  elicited  from  personnel.  The  “And  Merge  With  Wait”  waits  for  all  the  inputs  to 
arrive  before  proceeding  to  the  next  event.  The  “Or”  icon  uses  Boolean  logic  to  proceed  if  either 
of  the  inputs  occur.  The  “Decision  Point”  output  is  based  on  the  likelihood  of  an  event  to  occur. 
The  probabilities  of  each  event  are  entered  into  the  properties  of  the  icon. 


30 


ExtendSim  Icon 


Function  Name 


p4364] 

Clinger-Cohen  Assess/Cert 

Event/Activity 

B  [2336]  J 

p  ^ 

Bring  3  processes  together  PreB 

And  Merge  with  Wait 

II  r[3184]M, 

Or 

rT4317ir39U 

Decision  Point 

SDR  2 

■^657^^ 

Submodel 

PDR  submodel 

Figure  11:  ExtendSim  Icons 

3.3.3  Phase  3 

The  third  phase  focused  on  determining  how  to  update  ExtendSim®  software  code  from 
ERAM  1.0  in  order  to  develop  the  coding  implementation  plans  for  ERAM  1.1,  1.2,  2.0  and  2.1. 
The  research  team  identified  required  updates  in  collaboration  with  the  SMC/XR  Aerospace 
team  by  providing  model  design  inputs  via  Microsoft  Visio  diagrams,  Microsoft  Word 
documents,  teleconferences,  in  person  meetings,  plain  text  and  ExtendSim®  drawings. 

For  ERAM  1.1,  while  transferring  the  model  from  Arena®  to  ExtendSim  ,  the  Aerospace 
Corporation  and  the  research  team  identified  several  portions  of  the  model  (specific  activities) 
that  didn’t  apply  to  space  requirements  and  acquisitions  or  activities  that  hadn’t  been  included. 
Recommended  coding  changes  were  diagramed  and  validated  by  referencing  the  sources 
reviewed  in  phase  one. 
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ERAM  1.2  design  updates  were  identified  by  analyzing  the  updates  that  occurred  between 
the  March  2003  and  December  2008  versions  of  the  DoDI  5000.02.  Additionally,  design 
updates  incorporating  guidance  from  the  DTM  09-025  SSAP  were  included. 

ERAM  2.0  further  improved  the  fidelity  of  the  model  by  establishing  parameters  for  the 
model  that  would  adjust  the  outputs.  Essentially,  these  are  the  settings  or  “knobs”  that  can  be 
changed  prior  to  running  the  model  and  include  factors  such  as  SPO  Experience  Level,  ACAT 
level  (for  space  programs,  almost  always  ACAT-1D),  TRL  and  what  JCIDS  documents  may  or 
may  not  exist.  To  model  the  human  factors  influence  on  a  program,  a  SPO  Experience  Matrix 
was  created  that  included  acquisition  experience  level,  resource  availability,  staff  experience, 
external  program  support,  etc.  The  user  input  of  this  matrix  globally  impacted  certain  ERAM 
activity  distributions.  The  details  of  this  will  be  discussed  further  in  Chapter  IV. 

The  final  version,  ERAM  2.1,  incorporated  the  largest  software  update.  This  increased  the 
fidelity  of  ERAM  1.0  by  identifying  and  incorporating  the  activities  that  occur  early  in  the 
capability  development,  to  include  user-selected  JCIDS  concepts  options  (following  output  of 
AFSPC  IPP  e.g.  paper  concept,  Advanced  Technology  Demo  (ATD),  Joint  Urgent  Operational 
Need  (JUON,  others),  as  well  as  JCIDS  and  DP  activities  preceding  MDD.  Information  for  this 
update  was  gleaned  from  Chairman  of  the  Joint  Chief  of  Staff  Instruction  3170.01G  (CJCSI 
3170.01G),  CCTD  Guide,  the  AoA  Guide,  Early  Systems  Engineering  Guide  and  Pre-MDD 
Handbooks  as  well  as  through  iterative  discussions  with  SMEs  located  at  AFSPC/A5  and 
SAF/AQRE.  Document  reviews  guided  the  first  draft  of  an  activity  diagram  representing  the 
flow  of  the  early  activities.  Subsequent  iterations  of  the  diagram  were  updated  based  on 
feedback  from  AFSPC/A5  and  SAF/AQ  SMEs  to  validate  the  activity  flow.  Additionally,  the 
SMEs  assisted  in  defining  timelines  for  activities  within  the  model  for  most  likely,  worst  and 
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best  case  timelines  for  the  activities  as  well  as  the  decision  point  probabilities  based  on  historical 
experience.  Another  valuable  source  of  data  was  the  Requirements  and  Management  Plan 
(RAMP)  initiative  developed  by  AFSPC.  This  was  a  recently  developed  work  breakdown 
structure  (WBS)  that  identified  all  activities  at  AFSPC/A5  and  was  a  useful  tool  for  identifying 
best  case  timelines  or  policy-directed  timelines  for  various  discrete  events  within  the  JCIDS  and 
acquisition  processes.  The  AF/A5  website  was  also  referenced  and  provided  estimated 
coordination  and  approval  timelines  for  achieving  a  Joint  Requirements  Oversight  Council 
(JROC)  approved  ICD. 

3.3.4  Phase  4 

This  phase  involved  implementing  the  ExtendSim®  code  design  changes  developed  in 
phase  three  by  leveraging  an  established  collaborative  working  relationship  between  the  research 
team  and  the  SMC/XR  Aerospace  software  engineers.  With  the  required  design  changes 
developed  in  the  previous  phase,  the  software  engineers  made  the  necessary  programming 
changes  to  ERAM  1.0.  Various  interactions  and  follow-up  conversations  were  made  to  clarify 
programming  inputs.  After  clarifying  certain  aspects  of  the  code,  the  teams  collaborated  to 
provide  one  another  feedback.  When  code  was  implemented,  it  was  released  for  review  with 
clarification  questions.  The  research  team  answered  the  appropriate  inquiries  based  upon  policy, 
guidance,  and  interview  notes. 

3.3.5  Phase  5 

At  the  conclusion  of  the  effort  of  the  first  four  phases,  the  results  were  captured  in  this 
research  report.  It  includes  notes  from  the  interviews,  information  from  the  policy  documents, 
elements  of  the  programming  code  as  well  as  general  observations/opinions  about  space 
capability  development.  During  the  interviews,  the  researchers  took  notes,  transcribed  them,  and 
then  coded  them  or  extracted  common  themes  to  incorporate  into  the  report.  Additionally,  the 
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literature  review  revealed  other  pertinent  themes  which  were  then  incorporated.  Results  of  the 
programming  updates  were  included  in  their  appropriate  sections. 

Another  aspect  of  the  research  included  reviews  of  other  acquisition  modeling  activities.  In 
the  socialization  of  this  research  with  various  organizations,  similar  but  not  duplicate  modeling 
efforts  were  discovered  some  of  which  were  inspired  by  the  methodology  used  by  Wirthlin 
(2009).  Many  of  the  different  developments  were  not  known  by  the  other  parties.  The  following 
models  and/or  tools  were  reviewed:  1)  Acquisition  Process  Model  (APM) 
(http://acpo.crethq.com/acpo.htm);  2)  Acquisition  Document  Development  and  Management 
(ADDM);  and  3)  RAMP.  Further  discussion  of  these  is  in  Chapter  V. 

3.4  Assumptions 

Development  of  space  based  technologies  is  outlined  in  the  AFSPC  Instruction  61-101 
(AFSPCI  61-101),  Space  S&T  Management  (18  October  2007).  This  instruction  documents  the 
process  of  identifying  and  prioritizing  which  space  science  and  technologies  are  worthy  of 
investment  in  order  to  reach  future  strategic  objectives  and  is  executed  through  the  AFSPC  IPP. 
Figure  12  provides  a  summary  of  the  activities  flows  of  this  process. 
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Include  robust  solution-finding  process  to  support  PEO  TEO  findings  (incorporates  concepts  from 
proposed  hinter  gatherer  process).  SMC  CC  approved  AS  9  as  a  principal  in  PEO  TEO 
Closed-Loop  Feedback:  Solutions  identified  by  PEO  TEO  are  approved  through  S&T  corporate  process  to 
gain  AFSPC  CC  approval  prior  to  being  implemented  by  AFRL 
AFRL  implementation  leverages  a  broad  set  of  tools  to  reduce  technology  risk  or  improve 
schedule  performance  attnbutes  of  required  technologies — using  them  to  address  PEO  TEO  issues  in  a 
structured  format  is  new.  Some  of  these  implementation  options  flow  through  AFSPC  S&T  process 
programs.  Small  Business  Innovative  Research  Independent  Research  and  Development  (SBIR  IRAD) 
captured  in  $&T  via  feedback  mechanism 

AFRL  Focus  Long  Term  Challenges  (FLTC)  address  technologies  beyond  the  scope  of  current 
developmental  engineeringplans — this  supports  HQ  AFSPC  s  long-range  plans  where  SMC  has  not 
provided  detailed  concepts  technology  requirements 

SMC  forum  to  refine  technology  requirements  from  the  System  Program  Director  (SPD)  -  supports  S&T 
guidance  development  at  HQ  AFSPC.  Opportunity  to  influence  more  than  just  the  space  portfolio 
SAT  Guidance  is  the  integration  of  all  MAJCOM  direction  into  the  framework  of  the  Commander's 
strategic  vision  and  goals,  including  management  and  process  implementation  functions 
S&T  corporate  process  allows  AFSPC  CC  intent  of  HQ  vector  on  all  issues  sent  to  external  agencies. 

All  existing  activities  in  the  command  receive  feedback  and  use  it  in  the  day-to-day  activities 


Figure  12:  Major  S&T  Process  Elements  (AFSPC  2007) 

Since  the  IPP  is  continually  evaluating  and  selectively  funding  specific  S&T  activities  that 

support  identified  Mission  Area  Architecture  gaps  and  occurs  in  advance  of  concept 
consideration,  this  was  not  included  in  ERAM.  It  is  a  very  important  process,  but  doesn’t  fall 
within  the  scope  of  the  objectives  of  this  research. 
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3.5  Limitations 


Despite  research  efforts  to  identify  all  sequential/parallel  activities  that  occur  early  in 
capability  gap  analysis  through  MS-C  of  a  program  and  the  decision  probabilities  at  the  various 
junctions,  as  with  all  models,  there  are  limitations  to  ERAM.  Some  of  these  are  generic  to  all 
models  while  others  are  specific  to  ERAM.  For  example,  modeling  personalities  and  human 
behaviors  is  extremely  difficult.  Personalities  can  have  a  significant  impact  on  interpersonal 
relationships  which  can  have  influence  on  the  success  of  a  program.  If  the  PM  has  a  positive 
relationship  with  the  MDA,  the  likelihood  of  a  positive  review  may  be  higher.  This  is  highly 
subjective  and  a  challenge  to  quantify  in  software  code. 

1)  Generic  modeling  limitations  include  scope  and  schedule  risks.  The  number  of  all 
the  events  of  the  acquisition  and  requirements  processes  could  lead  to  several  thesis  and 
dissertations  to  improve  model  robustness.  The  researchers  needed  to  limit  the  focus  areas  and 
allow  for  future  research  to  continue  improving  ERAM.  The  schedule  for  this  project  was 
bounded  by  immovable  schedule  deadlines.  Heldman  notes:  “When  none  of  the  constraints  are 
negotiable — watch  out”  (Heldman  2005).  Therefore,  the  scope  had  to  be  clear  and  adjustable  as 
deadlines  grew  near  and  understanding  of  the  model  increased. 

2)  A  significant  common  factor  in  all  activities  within  capability  analysis,  requirement 
development/validation  and  executing  the  acquisition  program  is  that  all  are  heavily  influenced 
by  the  personnel  involved  on  the  program.  This  is  the  human  factors  element  that  is 
extraordinarily  difficult  to  accurately  capture  in  the  model.  A  sampling  of  these  issues  are 
identified  in  Table  3. 
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Table  3:  Human  Factors 


Human  Factor 

Characteristic 

Impact 

Military  Personnel 

Turnover  ~2-4  years 

Loss  of  technical/contract/ 
program  knowledge  negatively 
impacting  program 

Elected  Officials 

Turnover  ~2-4  years 

Funding,  strategic  prioritites 

Key  Personnel 

Leadership/Management/ 
Interpersonal  Relationships 

PM,  CO,  and  JAG  can  each 
individually  have  drastic 
impacts  on  program 
performance 

Senior  Leaders  (OSD,  MDA, 
PEO/SP,  AFSPC/CC, 

AFSPC/A5,  SAF/AQ) 

Turnover  ~2-4  years 

Program  priority  resulting  in 
change  in  available  resources, 
sponsorship 

Govt  support  contractors 

Quality  and  Technical 
Competency 

Experience  and  qualifications 
can  result  in 
insufficient/inadequate 
program  support 

Federally  Funded  Research  & 
Development  Center  (FFRDC) 
Contractors 

Availability 

Insufficient  technical  support 
impedes  government’s 
independent  assessments 

Industry  Personnel 

Management  &  Technical 
Performance 

Profit  versus  schedule  and  cost 
prioritization 

For  instance,  military  personnel  are  rotating  every  2-4  years.  Space  programs  can  easily 
last  ten  to  fifteen  years,  implying  there  will  be  a  significant  turnover  at  the  SPO  all  the  way  up  to 
DoD  senior  decision  makers.  This  turnover  in  personnel  can  be  either  very  beneficial  or 
detrimental  to  the  original  schedule,  but,  in  either  case  adds  significant  variability  to  ERAM 
results.  Certain  key  personnel  positions  in  the  requirements  and  acquisition  system  can 
significantly  influence  ERAM  outputs.  For  example,  in  the  acquisition  system,  the  PM  and 
contracting  officers  (CO)  are  critical  positions  to  the  execution  of  the  program.  The  PM  position 
is  competitively  managed  by  Air  Force  Personnel  Command  (AFPC)  as  a  Materiel  Leader. 

Strict  qualification  requirements  are  levied  and  a  board  of  senior  Air  Force  officers  selects  from 
qualified  candidates  to  fill  the  positions.  On  the  contrary,  the  CO  is  more  often  than  not  filled  by 
who’s  available  at  that  location;  they  are  essentially  a  high  demand,  low  density  Air  Force  asset. 
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The  CO  may  or  may  not  be  technically  competent  or  have  past  performance  qualifications  to 
successfully  support  the  schedule  and  cost  constraints  of  the  program.  Common  feedback  from 
interviewed  PMs  was  that  a  CO’s  “can  do”  attitude  or  “can’t  do”  attitude  can  critically  impact 
schedule  objectives.  Additionally,  the  performance  of  industry  personnel  were  difficult  to 
model.  Depending  on  the  profitability  of  a  program,  it  may  or  may  not  have  the  “all-star” 
personnel  assigned  to  it. 

ERAM  2.0  attempts  to  capture  the  impacts  of  these  human  factors  by  having  a  parameter 
that  can  be  adjusted  based  upon  the  composition  of  the  team.  However,  this  does  not  fully 
capture  the  variability  that  people  contribute  to  the  system.  The  success  or  failure  of  a  program 
is  greatly  impacted  by  the  levels  of  education,  motivation,  acquisition  experience,  and 
personality  of  all  team  members,  both  on  the  government  and  industry  teams.  The  column 
headings  below  relate  to  the  human  factor  indicators  in  Table  3  as  they  impact  the  variables  in 
experience  and  qualifications  of  the  program  office  team. 


Table  4:  Acquisition  Maturity  Potential  Matrix 


Acquisition  Maturity  Potential  Matrix 

Senior  Leadership 
Experience  in  position 

Staff  Experience  in 
position 

Senior  Leadership 
Cohesiveness 

Staff  Certifications, 
Training  and 
Motivations 

External  Program 
Support 

Program  Office  Size 

Level  0 

Has  less  than  1  year 
experience 

Has  less  than  1  year 
experience 

Has  not  worked 
together 

Staff  has  minimal 
Acquisition  training;  few 
if  any  certifications 

No  interest  beyond 
MAJCOM 

No  formal  program 
office  yet 

Level  1 

Has  1  to  3  years 
experience 

Has  1  to  3  years 
experience 

Has  worked  together  for 
less  than  6  months 

Staff  has  some 
Acquisition  training  & 
Certifications 

Some  interest 

Less  than  50%  of 
authorized  staffing 
Level 

Level  2 

Baseline 

Has  3  to  5  years 
experience  with  Acq 
processes 

Has  3  to  5  years 
experience  with  Acq 
processes 

Has  worked  together  for 
1  year 

All  Staff  has  some 
Acquisition  training  & 
Level  1  Certifications 

Senior  leadership 
helping  with  Acq 
process 

Between  50%  and  70% 
of  authorized  staffing 
Level 

Level  3 

Has  years  of  Acquisition 
Experience 

Has  years  of  Acquisition 
Experience 

Has  worked  together  for 
1  to  3  years 

Motivated,  certified 
(some  Level  II)  and 
trained  staff 

Senior  leadership  wants 
program  to  go 

Greater  than  70%  of 
authorized  staffing 
Level 

Level  4 

Has  been  steering 
program  through  Acq 
Process 

Has  been  steering 
program  through  Acq 
Process 

Has  worked  together  for 
3  to  5  years  but  not 

more 

Highly  motivated  and 
trained  staff 
(Certifications  Levels  1, 
II,  and  III  as 
appropriate) 

Senior  leadership  and 
congressional  interest 

Fully  staffed  to 
authorized  staffing 
Level 

ERAM  2.0  adds  a  global  variable  of  acquisition  capability  into  ERAM  1.2.  Table  4  identifies 
areas  where  an  ERAM  user  can  identify  the  appropriate  level  to  reflect  the  capabilities  of  the 
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SPO.  Appendix  B  documents  the  activities  selected  that  will  be  impacted  by  this  global 
variable.  This  variable  has  levels  from  0  to  4  which  modify  the  triangular  distribution  impact  for 
the  timelines  of  the  discrete  events. 

3)  Challenges  in  modeling  the  activities  in  the  requirements  process  are  found  where  the 
urgency  for  a  capability  receives  various  levels  of  advocacy  from  senior  leadership.  If  a  program 
is  influenced  a  certain  way,  some  of  the  modeled  process  could  be  waived  or  bypassed. 

However,  through  this  research,  no  distinguishable  pattern  was  found  for  waiving  processes. 
Leadership  may  change  which  increases  or  decreases  momentum  behind  a  program.  Programs 
could  “sit  on  hold”  for  years  while  different  political  processes  chum.  On  the  other  hand,  urgent 
needs  could  enter  the  process  at  varying  points.  Although  the  rapid  capability  acquisition 
framework  was  designed  for  ERAM,  further  research  is  required  to  implement  this  process. 

Even  when  it  is  implemented,  there  will  remain  a  level  of  uncertainty  as  to  how  personalities  will 
influence  this  unique  process. 

4)  This  research  approach  did  not  add  any  further  PPB&E  activities  to  ERAM  1.0. 
Although  there  was  a  cost  growth  check  during  technology  development,  no  further  events  were 
added  to  the  model.  This  calendar  driven  process  was  also  simulated  at  a  certain  level  in  ERAM 
1.0  and  to  add  further  detail  to  that  model  would  be  outside  the  scope  of  this  effort.  It  is  left  for 
further  research  to  include  higher  fidelity  modeling  in  this  category. 

5)  The  probability  of  the  various  triangular  distributions  on  event  timelines  in  later 
versions  of  ERAM  is  based  on  a  small  sample  size  due  to  the  limited  number  of  available 
personnel  for  interviews.  Schedule  constraints  also  limited  access  to  SMEs  in  certain  areas. 
Further  research  should  increase  the  sample  base  and  improve  the  fidelity  of  these  distributions. 
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6)  OSD  has  the  authority  to  issue  a  Resource  Management  Decision  (RMD)  to  a  SPO 
directing  specific  actions  for  their  program.  This  can  occur  at  any  time  during  a  program.  Since 
this  is  such  an  unpredictable  event,  this  will  be  left  for  further  research  to  determine  the  best 
model  entry  points.  Examples  of  the  RMD  could  include  specifying  an  acquisition  strategy, 
downward  directed  contracting  methods,  or  other  directives,  all  of  which  could  significantly 
impact  the  timeline  to  reach  MS-C. 

7)  PMs  and  other  SPO  personnel  may  have  irregular  tasking  or  requests  from  outside 
agencies.  These  events  occur  outside  the  regular  flow  of  the  process  model.  These  irregular 
“firefighting”  activities  that  acquisition  personnel  accomplish  are  partially  modeled  in 
uncertainty  events  which  were  included  in  the  original  ERAM  and  are  also  explained  further  in 
this  research  paper.  However,  due  to  the  unpredictability  of  these  events,  they  are  not  modeled 
explicitly  for  every  possibility.  The  number  of  permutations  or  possibilities  is  too  large  to 
accomplish  in  this  effort.  The  actual  performance  of  any  process  depends  on  two  factors:  the 
amount  of  time  spent  working  and  the  capability  of  the  process  used  to  do  that  work  (Repenning 
and  Sterman  2001).  Figures  13  through  15  below  shows  how  there  is  a  balance  needed  between 
spending  resources  improving  the  process  and  the  time  spent  working  on  the  current  activities. 
Two  approaches  are  “Working  Harder”  and  “Working  Smarter”. 
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Figure  13:  Work  Harder  Balance  Loop  (Repenning  and  Sterman  2001) 

Figure  13  describes  a  process  where  an  organization  focuses  on  the  current  activities  or  “fires”. 

Training  and  other  improvement  efforts  are  put  on  hold.  In  the  Work  Harder  loop  as 
performance  expectations  are  not  met,  pressure  to  produce  more  increases  which  lengthens  the 
time  spent  working.  However,  as  an  individual’s  time  is  maxed  out  working  the  current  issues, 
this  model  demonstrates  that  long  term  performance  decreases. 
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Figure  14:  Work  Smarter  Balance  Loop  (Repenning  and  Sterman  2001) 

Figure  14  shows  a  cycle  of  competing  demands  of  working  current  issues  and  the  pressure  to 

improve  overall  capability.  When  the  desired  performance  doesn’t  meet  the  current 
performance,  time  is  invested  in  improving  overall  capability.  If  too  much  emphasis  is  placed  in 
improving  capability,  actual  performance  decreases.  If  too  much  time  is  spent  on  improvement, 
resources  are  removed  from  actual  current  work  requirements. 
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Figure  15:  Simulations  of  Working  Harder  and  Smarter  Strategies  (Repenning  and 

Sterman  2001) 

Care  is  needed  in  determining  this  balance.  As  seen  in  Figure  15,  when  the  majority  of  the 
emphasis  is  spent  on  working  current  “fires”  or  issues,  there  is  a  net  decrease  in  capability  and 
performance.  When  there  is  a  balance  of  time  spent  improving  capability,  there  is  a  short  term 
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decrease  in  performance  but  it  rebounds  into  an  overall  net  increase  in  both  capability  and 
performance  on  the  current  tasks.  As  this  balance  is  achieved,  the  uncertainty  resulting  from 
“firefighting”  should  reduce  and  predictability  in  productivity  should  increase.  As  the  ever 
increasing  demands  on  personnel  in  DoD  acquisitions  coupled  with  personnel  decreases,  it  is 
challenging  to  anticipate  a  situation  where  it  will  be  easy  to  strike  the  balance  as  military 
personnel  only  remain  in  their  positions  for  a  short  period  of  time.  If  they  invest  in  capability 
improvement,  the  net  increases  in  capability  will  not  be  realized  during  their  tenure.  In  fact,  if 
they  implement  process  improvements,  they  may  be  in  the  short  term  productivity  “well”  shown 
in  Figure  15  (top  right  graphic).  Understanding  leadership  and  applying  these  human  factors  are 
considerations  in  modeling  the  acquisition  process  for  ERAM. 

8)  The  capability  gap  analysis,  requirements  development,  JCIDS  process  and  acquisition 
system  change  significantly  every  10  years  or  less.  The  names  of  documents,  processes,  reviews, 
and  others  change  frequently.  Therefore,  to  find  historical  data  upon  which  the  outputs  of 
ERAM  are  based  upon  was  extraordinarily  challenging,  especially  for  space.  Space  programs 
take  10-15  years.  Due  to  budget  constraints,  it  appears  that  new  program  of  record  starts  will  be 
very  rare.  Current  programs  of  record  will  be  modified  and  upgraded  in  place  of  new  starts. 

Thus  the  “fuzzy  front  end”  of  the  requirements  process  could  be  iterations  of  existing  programs 
returning  to  earlier  parts  of  the  acquisition  process. 

9)  Most  importantly,  it’s  necessary  to  accept  the  fact  that  every  capability  development 
program  is  unique!  Basing  the  development  of  a  new  capability  on  historical  data  is  inherently 
flawed  since  the  capability  has  never  been  developed  and  there  are  an  infinite  amount  of 
unknown  unknowns.  ERAM  generates  a  histogram  based  on  data  gathered  from  SMEs  in  space 
capability  development,  but,  there  is  always  the  chance  for  an  outlier.  Just  as  the  prediction  of 
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future  stock  prices  on  the  financial  markets  is  impossible  to  do,  so  is  the  exact  prediction  of  the 
schedule  for  defense  capability  development.  ERAM  is  a  tool  to  help  scope  the  effort  of  the 
problem,  but,  in  no  way  can  guarantee  what  the  realized  schedule  will  be. 
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IV.  Results  and  Analysis 


4.1  Phase  1:  Information  Collection  &  Review 

4.1.1  Interviews 

The  researchers  obtained  information  through  interviews  of  SMEs  in  the  space 
development  planning,  requirements  and  acquisition  fields.  The  interviews  revealed  opinions 
about  problems  with  existing  processes  and  policy  from  personal  experience  as  well  as  specific 
experiences  with  process  activities  and  associated  timeframes.  In  an  effort  to  consolidate  and 
filter  the  large  quantity  of  interview  data,  the  data  was  coded  into  common  themes  resulting  from 
the  conversations.  Twenty-three  SMEs  were  contacted  from  SAF/AQ,  United  States  Strategic 
Command  (USTRATCOM),  AFSPC,  and  SMC.  Table  5  shows  these  themes  in  order  from  the 
most  frequently  discussed  in  these  interviews.  Interpretation  of  this  qualitative  data  resulted  in 
observations  and  recommendations  for  ERAM. 

Table  5:  Research  Interview  Themes 


Interview  Themes 

%  Discussed 

1) 

SPO/MAJCOM/HHQ  Interpersonal  Relationships  &  PM  Credibility 

74% 

2) 

Direct  Higher  Headquarter  (HHQ)  Involvement 

48% 

3) 

Requirements  “creep” 

30% 

4) 

PM  Personnel  Requirements 

17% 

5) 

DP  Funding 

13% 

6) 

Requirements  personnel  Training 

13% 

7) 

Senior  Leadership  Direction 

9% 

8) 

Process  Flexibility 

9% 

1)  In  discussions  with  PMs  at  SMC,  some  challenges  identified  in  the  process  were  that 
the  MAJCOM  had  to  have  all  their  “i’s”  dotted  and  “f  s”  crossed  until  they  would  let  users  or 
HHQs  get  involved.  This  “coordination  in  a  vacuum”  proved  cumbersome  and  introduced 
delays  into  the  JCIDS  process.  When  the  PM  was  attempting  to  meet  demands  of  leadership  and 
perform  a  rapid  acquisition,  their  experience  was  that  the  MAJCOM  was  not  a  willing  partner 
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with  the  goal  of  speeding  up  the  process.  The  PM  related  an  analogy  that  the  command  lead 
was  analogous  to  a  pet  dragon.  He  would  need  to  be  regularly  fed,  but  had  the  capability  to 
“breathe  fire”  on  the  program  meaning  the  ability  to  significantly  impede  program  progress. 

The  PM  had  a  program  to  execute  with  milestones  and  deadlines  to  meet,  but  this  “fire  breathing 
dragon”  was  a  matter  of  bureaucracy  difficult  to  overcome.  HHQs  was  not  receiving  regular 
communication  due  to  this  challenge  at  command.  PMs  were  working  on  balancing  relationships 
with  their  command  leads  and  HHQ.  With  the  programs  limited  on  staffing  and  funding  at  this 
level,  they  didn’t  have  the  schedule  margin  or  breathing  room  to  spend  enough  time  thinking 
about  the  program’s  strategy. 

Another  PM’s  perspective  was  that  the  credibility  of  the  PM  is  what  determined  the 
success  of  the  program.  By  establishing  credibility  through  continuous  interactions,  constant 
meetings  via  teleconference  or  in  person,  the  program  was  able  to  maintain  a  continuous  flow  of 
communication  and  obtain  approval  on  required  documentation  in  a  timely  manner.  Contrary  to 
what  was  expected,  this  classified  program  completely  followed  the  JC1DS  and  DoDl  5000.02 
acquisition  policy  without  seeking  waivers.  The  program  followed  guidance  from  Dr.  Ashton 
Carter’s  memo  dated  14  Sep  2010  for  greater  efficiency  and  productivity  in  defense  spending. 
This  guidance  included  initiatives  on  affordability,  improved  cost  management,  and  improved 
incentivizing  of  the  industry  partners.  Due  to  the  successful  implementation  of  these  measures, 
the  credibility  of  the  PM,  and  continuous  communication  practices,  this  program  successfully 
completed  the  Defense  Acquisition  Board  (DAB)  in  a  record  41  minutes.  Unlike  many  other 
space  programs,  this  program  had  the  luxury  of  hand  selecting  team  members  from  the  SPO  to 
achieve  an  “all-star”  team.  The  relationship  with  the  SPO  and  the  Program  Element  Monitor 
(PEM)  was  ideal.  Each  party  understood  their  lanes  of  the  road.  Vigilant  leadership  established 
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credibility  and  a  well  established  line  of  communication  with  HHQs  enabling  this  remarkable 
achievement. 

According  to  other  senior  leaders  and  systems  engineers  at  the  center,  the  proposition  that 
“acquisition  is  a  team  sport”  was  reinforced.  The  forging  of  relationships  and  constant 
situational  awareness  of  programmatics  among  all  levels  was  crucial  to  positive  feedback  and 
program  success.  Indeed,  acquisition  success  was  based  on  the  people,  their  ownership  in 
program  success,  and  the  continuous  flow  of  information. 

2)  A  PM  identified  where  a  RMD  was  issued  by  OSD  in  order  to  provide  direction  to  a 
program.  In  this  case,  the  RMD  was  issued  to  define  the  program’s  acquisition  strategy.  This  is 
an  OSD  document  to  direct  a  program  when  the  Program  Objectives  Memorandum  (POM) 
process  doesn’t  fix  problems  in  the  program.  It  “gets  overwhelming  fixing  problems  one  year  at 
a  time”.  Therefore,  the  process  of  maintaining  levels  of  communication  with  HHQs  helps  to 
alleviate  any  misinformation  with  respect  to  the  expectations  of  the  program. 

3)  Systems  engineering  has  received  a  large  amount  of  press  with  respect  to  its  ability  to 
make  or  break  a  program’s  performance.  A  challenge  identified  by  members  was  the  lack  of 
disciplined  configuration  control  of  requirements  and  a  clear  understanding  of  how  to  manage 
the  trade  space  between  overarching  system-level  requirements  and  derived  requirements.  It  is 
extremely  difficult  for  a  program  manager  and  chief  engineer  to  manage  the  technical 
performance  of  their  program  with  a  “moving  target”  of  requirements.  Even  when  key 
performance  parameters  (KPP)  are  defined  and  a  capability  development  document  (CDD)  is 
signed  and  delivered  to  a  SPO,  requirements  creep  tends  to  continue  to  manifest  itself  in 
programs.  These  are  not  formal  changes  to  the  CDD,  they  are  small  changes  such  as  e-mails  that 
don’t  get  reviewed.  The  quandary  is  in  how  much  direction  the  SPO  can  provide  the  contractor 
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within  the  scope  of  the  contract  without  going  through  an  often  timely  and  costly  formal 
engineering  change  process  (ECP).  There  needs  to  be  some  level  of  control  and  tracking,  but 
within  the  proper  balance. 

4)  One  PM’s  approach  to  balancing  programmatic  issues  was  to  model  the  Commander’s 
intent  and  delivered  a  PM’s  intent  similar  to  the  Joint  Operations  Planning  Process.  This 
delivered  the  message  and  maintained  a  focus  within  the  members  of  the  SPO.  His 
recommendation  was  to  limit  a  PM’s  team  to  no  more  than  40  people.  This  was  the  maximum 
amount  a  single  leader  could  have  to  be  effective  and  manage  the  “face  time”  of  the  subordinates 
to  provide  programmatic  direction. 

5)  The  S&T  community  within  the  DoD  also  weighed  in  on  the  acquisition  process. 

Many  good  ideas  are  developed  at  the  research  labs  in  basic  and  applied  research.  As  these  ideas 
grow  in  technological  maturity,  the  lead  developer,  whether  it  is  the  government  sponsor  or 
contractor  will  start  to  look  for  sponsorship  and  advocacy.  However,  it  is  extremely  difficult  in  a 
constrained  budget  environment  to  obtain  funding  for  new  ideas.  One  member  of  the  S&T 
community  had  been  advocating  for  a  program  that  maintained  its  priority  just  below  the 
MAJCOM  cut  line.  DP  is  never  included  in  cost  estimates.  The  Program  Element  (PE)  or 
budget  line  for  activities  prior  to  MS-B  is  not  consistently  funded  or  defended  during  the  PPB&E 
process  and  is  often  used  as  a  source  of  funds  to  pay  other  bills.  Therefore,  these  DP  activities 
require  sponsorship  from  an  organization  with  some  level  of  discretionary  funding. 

6)  Continued  conversations  with  MAJCOM  personnel  highlighted  the  results  of  changes 
and  areas  of  emphasis  from  the  A1P  dated  4  May  2009.  It  states:  “Requirements  must  be 
acquisition- friendly  and  produced  in  a  format  that  is  readily  adaptable  for  use  during  source 
selection  and  throughout  the  acquisition  process”.  The  MAJCOM  recognized  that  their 
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personnel  needed  to  be  trained  in  requirements  management  and  development.  Therefore,  they 
developed  a  requirements  certification  program.  This  created  a  uniform  methodology  to  grow 
requirements  leads  and  meet  the  intent  of  the  AIP. 

7)  AFSPC  senior  leaders  desire  the  ability  to  quickly  understand  the  impacts  a  single 
program  may  have  across  the  space  capability  spectrum.  This  requires  developing  and  actively 
maintaining  a  robust  space  systems  architecture.  As  capability  development  program  schedules 
deviate  or  requirements  are  modified,  being  able  to  identify  the  implications  is  currently  a  timely, 
costly  and  difficult  challenge,  at  best. 

One  example  of  a  challenging  aspect  of  modeling  the  acquisition  processes  is  the 
leadership  and  its  influence  on  the  program.  According  to  one  perspective,  “senior  leadership  is 
the  wildcard  in  the  process”.  One  program  was  tackling  the  multiple  organizations  vying  for 
opportunities  to  manage  it.  It  was  led  by  two  Program  Executive  Officers  (PEO)  responsible  for 
different  components  of  the  overall  program.  With  two  programs  reporting  to  different  chains  of 
command,  it  was  extremely  difficult  to  have  any  synergy  in  program  execution.  This  program 
had  an  approved  Operational  Requirements  Document  (ORD)  based  on  previous  acquisition 
guidance.  Nine  subsequent  CDDs  were  developed.  Five  years  after  ORD  approval,  the  program 
was  cancelled  by  SAF.  The  bottom  line  was  that  a  large  acquisition  program  should  not  be  split 
between  two  centers  and  two  PEOs  for  political  purposes. 

Although,  small  programs  in  theory  would  have  less  oversight  from  HHQ,  there  are  some 
exceptions.  Programs  that  fall  under  the  Joint  Capability  Areas  (JCA)  deemed  as  “JROC  Interest 
Items”  will  automatically  be  reviewed  at  the  highest  level.  According  to  one  requirements  lead, 
that  is  “overkill”.  The  JCAs  categorized  as  a  JROC  Interest  Items  include:  Command  and 
Control  (C2),  Net  Centric,  and  Intelligence,  Surveillance  and  Reconnaissance  (ISR).  Therefore, 
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JCIDS  milestones  for  a  small  ACAT  III  level  program  at  these  JCAs  would  have  several  more 
months  of  time  due  to  this  level  of  interest.  If  the  proposed  capability  is  a  JROC  Interest  Item, 
historically,  another  three  months  is  required  to  get  an  ICD  written  and  through  the  Air  Force 
Requirements  Oversight  Council  (AFROC).  Cumulatively,  an  additional  nine  to  twelve  months 
is  required  to  get  a  JROC  approved  ICD.  Of  note,  nearly  all  space  programs  fall  within  the 
JROC  Interest  Item  category. 

8)  One  criticism  of  the  JCIDS  requirements  model  was  the  lack  of  flexibility.  Those 
working  in  requirements  for  Information  Technology  (IT)  or  Cyber  were  confronting  the 
challenges  of  accomplishing  the  various  events  of  the  JCIDS  process  while  attempting  to 
perform  these  in  a  rapid  fashion  in  keeping  up  with  the  fast  moving  pace  of  IT.  Senior 
leadership  would  perhaps  question  the  slow  pace  of  meeting  JCIDS  milestones  in  this  cyber  or 
IT  environment.  For  example,  if  software  or  a  piece  of  standard  computer  hardware  needed  to  be 
upgraded  it  would  require  an  AoA  and  other  activities.  The  MAJCOM  leads  suggested  that  in 
the  IT  world,  with  the  requirement  to  perform  rapid  technology  refreshes,  all  of  the  JCIDS  steps 
should  be  accelerated  or  eliminated.  This  would  be  able  to  significantly  compress  the  timelines. 
The  AoA  could  be  replaced  with  a  business  case  analysis  which  could  be  completed  in  less  than 
a  month  with  a  small  number  of  personnel.  The  MAJCOM  lead  said  “I  don’t  want  the  process  to 
be  different.  It  needs  leadership  to  be  able  to  apply  flexibility.” 

JCIDS  Process  Inputs: 

Of  the  23  individual  contacts  12  were  able  to  provide  input  to  validate  ERAM  1.0  and 
provide  updated  information.  A  portion  of  the  interviews  was  spent  understanding 
organizational  roles  and  responsibilities.  From  AFSPC/A5,  it  is  SMC’s  responsibility  to  develop 
materiel  solutions  in  response  to  AFSPC’s  validated  shortfalls  and  gaps  that  are  discovered  in  the 
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CBA.  The  CBA  is  a  MAJCOM  responsibility  that  is  accomplished  by  an  0-6  led  Capability 
Team.  The  capability  teams  at  AFSPC  are  identified  in  Table  6. 


Table  6:  AFSPC  Capability  Teams  (AFSPC/CV  2010) 


Primary  Director 

Capability  Team 

A5 

Situational  Awareness/Command  &  Control  (SA/C2) 

A3 

Space  Protection  and  Information  Operations 

A5 

Missile  Waming/Missile  Defense  (MW/MD) 

A2 

Battlespace  Awareness/Technical  Intel  (BA/TI) 

A3 

Position,  Navigation  and  Timing  (PNT) 

A5 

MILSATCOM 

A3 

Launch,  Ranges  and  Networks 

A5 

Operationally  Responsive  Space 

A3 

Training,  Exercises  and  Evaluations 

A3 

Current  Operations 

A5 

Policy  and  Integration 

A8 

Total  Force  Integration  (TFI) 

A3 

Air  Expeditionary  Force  (AEF) 

A3 

Cyber  Warfare 

A3 

Cyber  Operations 

A6 

Cyber  Infrastructure 

A3 

Testing,  Modeling  &  Simulation 

A8 

Nuclear  Issues  and  Matters 

These  Capability  Teams  request  space  DP  efforts  from  AFSPC/ A5X  on  behalf  of  combatant 
commanders  (COCOM)  (i.e.  Warfighter).  AFSPC/A5X  has  a  30  day  timeline  to  develop  a 
proposal  and  provide  it  back  to  the  Capability  Team  for  approval.  In  addition  to  DP  activities 
from  the  Capability  Teams,  AFSPC/A5X  does  an  annual  AF  wide  data  call  for  DP  proposals  in 
early  November  (AFMC/A2/5  2010). 

While  accomplishing  the  DP  activity,  AFSPC  can  tap  into  technical  SMEs  through 
SMC/XR  or  directly  through  an  existing  SMC  SPO.  One  of  the  products  of  DP  from  SMC  is  the 
CCTD.  Ultimately,  this  early  analysis  focuses  on  determining  if  the  Materiel  Development 
Decision  (MDD)  Acquisition  Decision  Memorandum  (ADM)  should  direct  accomplishing  a  new 
Analysis  of  Alternatives  (AoA).  Depending  on  the  urgency  of  the  capability,  either  CYBER 
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SAFARI  (6-9  months)  or  traditional  (8-12  months)  DP  is  accomplished.  With  an  appropriate 
level  of  technical  analysis  and  early  systems  engineering  applied  during  the  CCTD  development, 
AoA  activities  should  be  reduced  from  18-24  months  to  just  6-9  months.  Of  note,  no  space 
AoAs  were  accomplished  in  FY10;  three  space  system  MDDs  are  scheduled  for  FY1 1. 

The  process  for  developing  a  CCTD  in  the  DP  process  is  initiated  by  the  MAJCOM. 
“Capability  Teams  will  meet  regularly  and  frequently  and  will  address  the  full  sets  of  issues 
(requirements,  trade-offs,  human  resources,  training,  infrastructure,  tactics  and  procedures)  in 
defining,  developing,  fielding  and  operating  a  new  capability.  Capability  Teams  will  be  major 
contributors  to  the  IPP  and  Corporate  Processes”  (AFSPC  2008).  After  an  ICD  was  complete 
through  the  IPP,  the  MAJCOM  A5  issued  a  DP  request  to  the  Center  XR  for  a  new  concept,  or  to 
the  Center  SPO  for  an  upgrade.  The  Center  is  given  30  days  to  provide  a  DP  Proposal  back  to 
the  MAJCOM.  This  DP  action  is  in  essence  the  development  of  a  CCTD.  These  CCTDs  can 
carry  single  or  multiple  concepts.  The  draft  CCTD  is  expected  approximately  8  months  after 
start  with  the  final  completed  by  12  months.  The  concepts  from  the  CCTD  studies  are  the 
“alternatives”  used  in  an  analysis  of  alternatives. 

The  document  administrator  at  HQ/AF  is  A5RP.  A5RP  provides  guidance  on  the 
standardized  package  that  should  be  submitted  for  HHQ  coordination.  Typically,  it  takes 
approximately  two  months  to  have  a  document  reach  the  A5RP  chaired  Requirements  Strategy 
Review  (RSR).  This  review  essentially  approves  the  pursuit  of  authoring  the  ICD  for  the 
capability  under  consideration.  Once  the  RSR  approves,  the  ICD  can  be  drafted  as  quickly  as 
one  week  by  the  High  Performance  Team  (HPT),  depending  on  the  urgency  and  available 
resources.  The  draft  ICD  can  be  entered  into  internal  (AFSPC)  and  external  (SAF  and  OSD) 
coordination  simultaneously  with  appropriate  AFSPC  senior  leader  approval  to  save  time. 
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Coordination  requires  a  minimum  of  three  weeks.  Upon  completion  of  coordination,  the 
document  will  be  reviewed  by  the  AFROC.  In  this  example,  the  AFROC  met  three  weeks  after 
coordination  and  comment  resolution.  Another  document  that  will  be  required  is  the  AoA  Study 
Guide.  Based  on  previous  examples,  in  this  case,  it  only  took  one  day  to  author  this  document. 
Of  note,  OSD/CAPE  is  responsible  for  approving  this  document  for  JROC  Interest  Items.  The 
example  that  was  discussed  in  the  interview  was  being  driven  by  AFSPC/A5  (Brigadier  General) 
and  was  definitely  unique.  Other  capability  documents  would  likely  take  three  times  as  long  to 
work  through  this  process. 

From  another  AFSPC/A5  source,  there  are  two  ways  capability  development  is  kicked 
off:  1)  downward  directed,  or  2)  determining  the  needs  of  the  warfighter.  The  most  timely  way 
to  kickoff  capability  development  is  to  tie  the  requirement  to  an  existing/approved  ICD, 
alleviating  many  of  the  early  JCIDS  activities  since  they  were  previously  accomplished. 

The  CBA  is  heavily  influenced  by  US  Strategic  Command  (USTRATCOM)/J8 
(representing  warfighter  needs),  however,  most  of  the  analysis  is  accomplished  by  AFSPC  with 
support  from  SMC.  A  CBA  for  a  revolutionary  capability  (new  system)  will  require  six  to 
twenty-four  months  while  one  for  an  evolutionary  capability  (upgrade  an  existing  system)  will 
take  approximately  three  to  six  months. 

From  another  AFSPC  interview,  the  ICD  requires  one  month  to  write  and  requires  six  to 
nine  months  for  coordination  and  approval.  The  ICD  may  result  in  no  action  or  kick  off 
preparation  for  MDD  and  ADM.  Policy  requires  that  funding  must  be  set  aside  to  reach  the  next 
milestone.  In  the  case  of  getting  an  ADM  from  the  MDA  at  the  MDD,  this  requires  funding 
through  MS-A. 
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Another  valuable  note  from  the  AFSPC  interviews  was  the  fact  that  either  AFSPC  or 


USTRATCOM  could  sponsor  an  ICD.  Should  the  combatant  command  sponsor  the  ICD,  this 
relieves  the  AFSPC  action  officer  from  Air  Force  specific  coordination  activities  (RSR  and 
AFROC). 

4.1.2  General  Comments/Other  Tools: 

ADDM: 

As  the  research  team  continued  to  detail  the  processes  and  activities  at  SMC,  AFSPC  and 
SAF/AQRE,  it  became  apparent  that  other  independent  efforts  with  goals  similar  to  that  of 
ERAM  were  underway  at  AFSPC  and  ASC/XR.  On  28  October,  2010,  Mr.  Blaise  Durante, 
SAF/AQX,  issued  a  memorandum  with  the  subject  of  “Acquisition  Document  Development  and 
Management  Initiative”.  This  was  distributed  to  each  of  the  Air  Force  PEOs  as  well  as  the 
Acquisition  Center  of  Excellence  (ACE)  at  SMC,  Aeronautical  Systems  Center  (ASC), 
Electronics  Systems  Center  (ESC)  and  Air  Armament  Center  (AAC).  It  summarized  the  purpose 
of  ADDM  and  directed  the  following  actions: 

1)  PEO:  Identify  the  priority  and  timing  for  each  of  your  program  offices  to  begin 
utilizing  ADDM  and  communicate  that  list  with  SAF/AQXI  in  order  to  establish  the 
deployment  and  support  schedule — complete  within  two  weeks  of  the  date  of  this 
memorandum; 

2)  PEO  and  ACE:  Begin  to  use  ADDM  to  identify  and  track  acquisition  milestone 
readiness  status  of  acquisition  programs  in  your  portfolios; 

3)  PEO  and  ACE:  Communicate  ideas  for  improvement  or  enhancement  of  ADDM. 
ASC/XR  has  been  made  aware  of  this  initiative  and  is  beginning  to  implement  this  tool 
(SAF/AQX  2010).  Unlike  ERAM,  this  tool  focuses  on  the  detailed  tasks  during  execution  and 
helps  provide  the  program  with  document  templates  depending  on  where  they  are  in  the 
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acquisition  process.  ERAM  is  a  modeling  tool  that  focuses  on  the  overall  process  and  providing 
early  schedule  analysis  data  to  decision  makers  prior  to  MDD.  Figure  16  shows  some  of  the 
capabilities  of  translating  DoDI  5000.02  into  an  executable  roadmap.  The  roadmap  is  similar  to 
Microsoft  Project®  to  layout  a  program  schedule.  The  SPO  is  then  able  to  tailor  the  roadmap 
and  document  templates.  ADDM  enables  the  planning,  tailoring,  development,  tracking,  review 
and  approval  of  milestone  acquisition  document  content  to  successfully  meet  the  next  milestone 
decision  date.  Additionally  it  provides  authoritative  references,  guidance,  and  instructions.  It  has 
the  capability  to  dynamically  generate  documents  based  on  most  current  data  in  system 
(ASC/XRCC  2010). 


Three  major  activities  need  to  be  performed  in  orderto  achieve  target  milestones. 
The  Roadmap  Manager  participates  in  each  of  these  activities 


II 

1. 

Create  Roadmap 

2. 

Roadmap  Setup 

• 

j  1 

TailorRoadmap 

• 

Create/Manage  Team  93 

• 

Initiate  DocumentScheduling 

**££3^* 

• 

Finalize  Roadmap  1 

3. 

Execute  Roadmap 

• 

Involves  all  team  members 

• 

Roadmap  Managermonitors  progress 

• 

Register  Milestone  Decision 

MiletlonfrB  MAT  1C 

. 

I  :l 

f - ™ 

■  I 

ul _ I 

LDlX  i II”- II  El  i 


“*■ 

BlUitnH* 


Simc 


2 


Fima?1*  wt  tom-i 


rsjKtsft*  -*rw  wc-s 


in 


1 


I  , , 


Figure  16:  ADDM  (ASC/XRCC  2010) 
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RAMP: 


AFSPC/A5  has  taken  the  initiative  to  develop  a  comprehensive  space  architecture  that 
will  enable  better  understanding  of  the  interdependencies  of  all  space  capabilities  and  the 
impacts  current  acquisition  programs  may  have  on  existing  and/or  planned  capabilities.  AFSPC 
acknowledged  the  need  to  identify  requirements  generation  process  improvements  to  bring  the 
“speed  of  need  to  space  &  cyber  acquisition”.  They  identified  nine  root  causes  to  these 
challenges  which  are:  procedures,  tribal  knowledge,  resources,  process  discipline,  monitoring, 
quality  assurance/quality  control,  requirements  definition,  poor  correlation  funding  and 
requirements,  and  systems  engineering  issues.  To  work  towards  overcoming  these  challenges, 
they  implemented  a  four-prong  approach  which  is:  process,  knowledge  infrastructure, 
organization,  and  people  (Gilchrist  201 1).  RAMP’s  development  contributes  to  these  solutions. 
To  accomplish  this,  the  first  step  A5  accomplished  was  to  develop  an  all  inclusive  WBS  structure 
of  all  activities  that  occur  at  both  AFSPC  and  SMC  when  developing  a  capability.  RAMP 
documents  requirements  and  acquisition  at  a  top-level  process  and  integrates  with  Microsoft 
Project®.  This  tool  references  guidance  and  policy.  It  allows  the  user  to  schedule  activities, 
track  resources  and  tailor  their  activities  (Gilchrist  2011).  Figure  1 7  shows  this  tailorable 
Microsoft  Project  ®  schedule  that  is  generated. 
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Figure  17:  RAMP  (Gilchrist  2011) 

APM: 

The  Acquisition  and  Excellence  and  Change  Office  (SAF/AECO)  developed  another 
process  model  of  the  acquisition  system.  It  is  called  the  Acquisition  Process  Model  (APM). 

The  intent  of  APM  provides  the  documented  current  state  process  in  an  interactive  fashion.  In 
2009,  the  ACPO  commissioned  DSD  Laboratories'  Center  for  Reengineering  and  Enabling 
Technology  (CRET)  to  develop  the  APM.  The  APM  is  a  compilation  of  policy,  instructions,  and 
guidance  for  persons  involved  in  the  acquisition  process.  The  APM  accomplishes  the  following 
goals:  Establishes  standard  definition  and  activities  associated  with  AF  Acquisition;  provides  an 
integration  context  for  other  extemal/related  process  models;  provides  the  process  input  to 
Acquisition  Enterprise  Architecture  and  other  Enterprise  Architectures;  provides  a  standard 
reference  model  for  all  stakeholders;  and  provides  a  common  context  for  process  improvement 
initiatives  (AF/ACPO  2011).  APM  is  an  interactive  tool  to  interface  with  various  portions  of  the 
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acquisition  “wall  chart”.  The  user  can  “click”  on  each  block  to  get  further  details  on  a  particular 
portion  of  the  acquisition  process.  Figure  18  shows  an  example  of  the  interactive  web  page. 
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Figure  18:  Acquisition  Process  Model  (ACPO  2011) 

Take-Away: 

Although  the  objectives  of  each  organization  and  their  respective  tools  are  unique,  the 
data  required  to  develop  the  respective  tools  may  be  shared.  For  example,  ERAM  is  a  simulation 
model  that  identifies  key  high-level  activities  and  associated  distributed  times  of  these  activities 
from  capability  gap  analysis  through  Milestone  C.  It  enhances  CCTDs  that  will  be  used  during 
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the  AoA  when  determining  which  Materiel  Solution  should  be  selected  and  pursued.  Similarly, 
RAMP  identifies  all  the  activities  and  a  specific  time  associated  with  each  activity  as  well. 
ADDM  is  an  acquisition  tool  that  provides  standardized  templates  of  required  acquisition 
documents,  of  which  will  be  required  at  various  points  in  ERAM  and  RAMP.  APM  provides  an 
interactive  repository  of  process  guidelines  and  steps  to  aid  acquisition  professionals  in  their 
understanding  of  the  acquisition  processes.  There  appears  to  be  great  potential  for  these  tools  to 
collaborate  together. 

4. 2  Phase  2:  ERAM  1 . 0  Analysis 
The  Model 

The  probabilistic  discrete  event  model  developed  by  Wirthlin  was  modified  with  a  space 
systems  focus  by  implementing  policy  change  updates  (post  2006)  and  enhanced  early  activities 
by  identifying  the  JCIDS  and  early  systems  engineering  activities  occurring  during  Material 
Solutions  Analysis  and  Technology  Development  phases.  The  PPB&E  process  can  significantly 
impact  the  execution  of  the  programs,  but,  due  to  the  dynamic  and  unpredictable  nature  of  the 
defense  budget,  few  updates  were  made  in  the  model  to  reflect  these  activities.  The  model 
attempts  to  account  for  the  budget  variability  by  utilizing  “funding  check”  nodes  at  a  variety  of 
places  throughout  the  model.  The  core  updates  to  the  model  were  divided  into  two  sections:  1) 
updates  to  DoDI  5000.02  since  ERAM  was  first  developed,  and  2)  additions  based  on  the  JCIDS 
process  specific  to  AFSPC. 
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4.3  Phase  3:  Identify  and  design  updates  to  ERAM  1.0 
ERAM1.1 

ERAM  1 . 1  focused  on  the  changes  required  due  to  porting  from  Arena®  to  Extends im©. 
The  Aerospace  Software  Engineers  audited  the  software  to  ensure  that  the  results  from  ERAM 
1.1  were  comparable  to  ERAM  1.0.  The  artifacts  of  running  a  new  simulation  software  program 
generated  a  small  number  of  changes.  These  specific  changes  were  implemented  and  updated 
into  ERAM  1.1  as  the  baseline  for  future  changes.  The  validation  and  verification  of  this  model 
is  being  conducted  wholly  by  Aerospace  and  is  outside  the  scope  of  this  effort. 

ERAM  1.2 

Policy  and  guidance  is  constantly  evolving  based  on  the  political  climate,  lessons-leamed 
from  existing  or  cancelled  programs,  funding  constraints  and  others.  The  changes  implemented 
in  ERAM  1 .2  reflect  space  acquisition  policy  updates  currently  being  incorporated  into  DoDI 
5000.02.  Previous  space  policy  documents  and  interim  guidance  have  been  rescinded  while 
policy  memos  are  serving  as  the  interim  guidance.  Additionally,  updates  to  DoDI  5000.02  since 
ERAM  1.0  are  being  incorporated  into  the  model.  Figure  19  describes  specific  DoDI  5000.02 
updates  to  include  the  preliminary  design  review  (PDR)  moving  prior  to  MS-B,  with  a 
placeholder  for  PDR  occurring  post  MS-B,  as  required. 
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Figure  19:  DoD  5000.02  Updates  (OUSD/ATL  2008) 

Figure  20  identifies  the  space  specific  acquisition  process  updates  incorporated  into  the 

2008  version  of  the  DoDI  5000.02;  ERAM  1.0  was  based  on  the  2003  DoDI  5000.02  version. 
ERAM  1 .2  added  MDD,  IPAs  and  Post  SDR  assessment.  These  IP  As  were  added  to  support 
MDA  decisions. 


62 


DoOl  5000.02  (0  Dec  2003  with  Space-Related  Rev  felons) 
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Figure  20:  Space  Acquisition  Policy  Updates  to  DoD  5000.02  (Skotte  2010) 

ERAM2.0 

ERAM  2.0  focused  on  incorporating  the  changes  from  the  SPO  capability  matrix.  This 
was  designed  by  instituting  a  human  factors  variable  to  be  implemented  as  a  global  variable  in 
various  timed  activities  in  ERAM.  This  was  designed  through  discussions  with  the  Aerospace 
design  leadership  and  the  researchers  to  determine  impacted  timelines.  It  was  based  on  the 
collective  understanding  and  knowledge  of  the  researchers  combined  with  learning  from 
readings  and  interviews.  Once  these  events  were  identified  they  were  delivered  to  the  design 
team  to  implement  in  ERAM  2.0.  Recall  that  Table  4  showed  the  factors  that  determine  the 
overall  experience  level  of  a  program  office  which  in  turn  changes  timelines  for  processes  in 
ERAM. 
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This  global  variable  skews  the  triangular  timeline  distributions  results  from  an  aggregation  of  the 
above  capabilities.  It  includes  the  number  of  years  the  senior  leadership  holds  a  position,  the 
years  of  experience  of  the  staff  members,  unit  cohesion  between  the  senior  leaders  and  their 
subordinates,  the  formal  certification  training,  support  from  external  organizations,  and  the 
percentage  of  filled  positions  in  the  program  office.  The  factors  contribute  to  the  aggregated 
global  variable  calculated  and  then  applied  to  the  impacted  process  activities  (see  Appendix  B). 
The  global  variable  has  levels  from  0  to  4  which  determine  how  the  triangular  distribution  of  for 
the  timelines  of  the  discrete  events  is  skewed. 

ERAM2.1 


Figure  21:  ERAM  2.2  Design 

Figure  21  illustrates  the  activity  diagram  developed  in  Microsoft  Visio  that  shows  the 
basic  activities  involved  between  capability  gap  analysis  up  through  conducting  an  AoA,  post 
MDD.  This  diagram  went  through  numerous  iterations  based  upon  feedback  from  both  AFSPC 
and  SAF/AQ  personnel  familiar  with  the  processes.  This  is  a  collaborative  representation  of  the 
processes,  as  everyone  interviewed  has  varying  opinions  how  S&T  research,  capability  gap 
analysis,  DP,  JCIDS,  CCTD  and  AoA  processes  occur  and  are  interrelated.  During  this  portion 
of  the  report,  the  activities  and  processes  will  be  discussed  and  how  they  are  intended  to 
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compliment  other  processes  to  deliver  new  defense  capabilities.  Additionally,  the  methods  to 
determine  the  triangular  distributions  of  time  required  for  each  activity  and  decision  point 
probabilities  will  be  discussed  as  well  as  the  results  of  the  analysis. 

Strategic  Discussion: 

Before  discussing  the  details  of  the  various  processes  and  organizational  relationships, 
it’s  beneficial  to  look  at  the  problem  from  the  strategic  level.  Two  common  philosophies  were 
discovered  in  discussions  with  personnel  familiar  with  defense  acquisitions  and  capability 
development:  technology  pull  and  technology  push.  The  first  is  when  the  warfighter 
(COCOMs)  identify  a  capability  gap  and  request  it  be  addressed  by  the  respective  MAJCOM — a 
reactionary  response.  The  second  method  occurs  when  industry,  universities  or  defense 
laboratories  proactively  market  technologies  and  their  potential  capabilities  to  the  MAJCOM, 
Joint  Staff  and/or  Air  Staff.  This  is  referred  to  as  technology  push  and  addresses  emerging 
threats  (both  theoretical  and  realized)  that  maturing  technology  may  address — the  proactive 
development  of  military  applications.  In  the  opinion  of  the  authors,  both  are  necessary,  however, 
technology  push  is  critical  for  the  United  States  to  continue  to  maintain  its  military  superiority 
against  all  adversaries. 

Expanding  upon  the  concept  of  technology  pull,  a  gap  has  been  identified  and  has  war 
fighter  sponsorship,  implying  congressional  and/or  flag  officer  interest.  Senior  leader 
sponsorship  is  critical  to  navigate  the  DoD  capability  development  process  since  there  are 
numerous  reviews  and  decision  points  where  a  concept  can  be  shelved.  The  higher  the  rank  of 
the  sponsor,  the  less  likely  it  will  be  shelved.  The  problem  with  this  method  is  the  JCIDS 
process  is  often  not  timely  enough  to  adequately  satisfy  the  urgency  of  the  need.  Hence, 

PPB&E,  requirements  management  and  acquisition  activities  often  become  jumbled  and  result  in 
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a  chaotic  work  environment  for  each  organization  involved.  Valuable  early  systems  engineering 
activities  are  frequently  rushed  or  neglected  due  to  the  time  and  cost  associated,  resulting  in 
decisions  being  made  on  inaccurate  or  insufficient  data  that  under  estimate  developmental  and 
delivery  costs  and  schedule. 

In  the  instance  of  technology  push,  this  embraces  innovation  and  addresses  emerging 
threats  and/or  capability  gaps — the  “what  ifs”.  This  method  typically  is  much  more  difficult  to 
find  senior  leader  sponsorship  since  it  is  competing  with  realized  threats  (tech  pull)  and 
admittedly,  has  challenges  making  it  from  the  research  facilities  into  the  JCIDS  process  to  the 
acquisition  centers.  In  theory,  during  technology  fairs,  demonstrations  and  exercises,  the 
MAJCOMs,  COCOMs,  Joint  Staff  and  Air  Staff  should  be  exposed  to  the  maturing  technology 
and  its  opportunities  for  military  application — marketing  opportunities  for  industry.  If 
successfully  marketed,  the  demonstrated  technological  advancements  and  their  potential  military 
application  will  trigger  initiation  into  the  JCIDS  process  by  the  MAJCOM. 

Furthermore,  technology  push  is  critical  for  industry  to  continue  to  grow  and  post  profits 
for  shareholders.  This  research  revealed  it  is  a  relatively  ungovemed  activity  with  lucrative 
potential.  To  take  advantage  of  this  opportunity,  industry  frequently  hires  retired  military 
personnel  with  acquisition  experience  as  “Business  Development”  managers.  These  persons  are 
typically  well  networked  within  DoD  staffs,  knowledgeable  on  the  capability  gap  analysis, 
requirements,  acquisition  and  PPB&E  systems  and  can  be  extremely  valuable  assets  to 
companies  since  they  are  considered  one  of  the  catalysts  that  triggers  the  initiation  of  new 
MDAPs. 

Having  discussed  the  two  philosophies  with  regards  to  how  emerging  technology  is  used 
to  enhance  existing  capabilities  and  develop  new  capabilities,  the  next  portion  of  the  report  will 
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discuss  the  pre-MDD  framework  that  was  established  for  ERAM  and  how  these  two 
philosophies  impact  it. 

Modeling  Capability  Gap  Analysis 

Significant  time  and  effort  was  spent  researching  and  understanding  how  the  desire  for  a 
new  space  capability  becomes  a  MDAP.  A  very  simple  and  basic  question  proved  challenging  to 
answer:  where  are  new  ideas  for  military  applications  created  and  how  are  they  formally  inserted 
into  the  DoD  requirements  and  acquisition  system?  There  are  both  informal  and  formal  S&T 
processes  supporting  AFSPC’s  responsibility  to  sustain  existing  capabilities  and  develop  new 
ones.  The  formal  process  will  be  discussed  first  followed  by  a  discussion  of  observations  on  the 
informal  methods.  Figure  22  circles  the  region  of  the  framework  under  discussion  in  this 
section.  It  was  necessary  to  understand  this  process,  however,  these  activities  and  decision 
points  were  not  modeled  in  ERAM  as  ERAM  models  an  idea  once  it  is  in  the  “system”. 
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Figure  22:  ERAM  2.1  Front  End  Design 

S&T  activities  are  formally  governed  by  Air  Force  Policy  Document  61-1  (AFPD  61-1), 
Management  of  Science  and  Technology,  Air  Force  Materiel  Command  Instruction  61-102 
(AFMCI  61-102),  Advanced  Technology  Demonstration  Technology  Transition  Planning,  and 
AFSPCI  61-101,  Space  Science  and  Technology  (S&T)  Management.  The  space  S&T  program 
encompasses  basic  research,  applied  research  and  advanced  technology  development  with  the 
goal  of  maturing  technologies  that  can  be  utilized  by  AFSPC  to  develop  materiel  solutions  and 
eventually  by  the  product  centers  to  develop  both  evolutionary  (upgrade  existing  systems  to 
provide  new  capabilities)  and  revolutionary  (new  systems  delivering  new  capabilities)  systems. 
Interestingly,  Air  Force  Materiel  Command  (AFMC),  not  AFSPC,  develops  space  S&T  POM 
recommendations  (with  inputs  from  Air  Force  Research  Laboratory  (AFRL)  and  each  of  the 
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product  centers)  for  investment  strategies  to  focus  on  developing  and  maturing  space 
technologies.  Of  note,  AFRL  is  subordinate  to  AFMC,  not  AFSPC — this  could  be  one  of  the 
reasons  many  of  the  interviewees  for  this  research  indicated  there  has  historically  been  a  problem 
with  maturing  space  technology  with  the  AFRL  and  transitioning  it  to  space  materiel  solutions 
analysis  and  MDAPs  at  AFSPC  and  SMC,  respectively.  This  organizational  structure  may  also 
have  limited  or  inhibited  the  robust  establishment  of  space  technology  expertise  within  AFRL. 
Senior  Air  Force  leaders  may  have  recognized  this  issue;  in  2010,  the  newly  appointed  AFRL 
commander  had  previous  assignments  as  the  Deputy  Director  of  the  National  Reconnaissance 
Office  (NRO)  and  prior  to  that  was  the  Vice  Commander  of  the  SMC.  But  this  remains  a 
piecemeal  fix  with  no  permanent  process  in  place  to  address  the  longer  term  issues. 

How  are  AFSPC  space  capability  and  technology  requirements  communicated  to  the 
S&T  community  (AFRL,  industry  partners  and  academia)?  According  to  AFSPCI  10-604,  the 
AFSPC  IPP  is  used: 


. .  .to  develop  an  executable  Investment  Strategy. .  .Through  the  IPP,  AFSPC  identifies, 
defines  and  prioritizes  needed  capabilities,  determines  shortfalls  that  must  be  filled 
through  modernization  and  transformation,  assesses  HIS  requirements,  and  develops  a 
fiscally  and  technologically  achievable  plan.  The  IPP  follows  phases  known  as  FAA  to 
determine,  categorize  and  prioritize  the  basic  functions  the  command  must  perform;  FNA 
that  identifies  and  prioritizes  the  capability  shortfalls;  FSA  to  assess  possible  materiel 
solutions  to  the  shortfall;  and  Integrated  Investment  Analysis  to  determine  the  optimal 
force  structure  given  resource  constraints.  Outputs  required  by  the  HQ  AFSPC/A3/A5  in 
support  of  this  AFSPCI  include  a  capabilities  needs  list  (shortfalls),  an  analysis  of 
solutions  and  some  initial  MAJCOM-level  FSA  data  for  JCIDS  and  the  results  of  IIA  that 
include  system  roadmaps  (out  year  funding  profiles,  IOCs,  EOLs  and  other  key 
milestones  (AFSPC/A3F  2007). 

One  of  the  products  of  the  IPP  is  the  Strategic  Master  Plan  (SMP).  This  document  is  intended  to 
provide  prioritized  guidance  to  the  labs  with  regards  to  what  emerging  space  technologies 
require  further  research  and  development  to  meet  the  long  term  capabilities  AFSPC  is  required  to 
provide  to  the  combat  commands. 
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After  reviewing  the  AFIs  and  AFSPCIs,  the  space  S&T  process  was  summarized  and 
shared  with  AFSPC  to  confirm  the  research  accurately  depicted  the  process.  Unfortunately,  their 
response  indicates  that  the  processes  outlined  by  the  instructions  are  not  current.  This  highlights 
a  major  problem  for  anyone  involved  in  S&T  and  capability  development.  Implications  include: 
1)  inability  for  formal  training  curriculum  to  be  developed  on  the  process  since  it’s  frequently 
changing,  2)  inability  for  outside  organizations  to  be  able  to  read  instructions  to  understand  other 
organizations  processes  resulting  in  degraded  inter-organizational  processes,  and  3)  numerous 
conflicting  interpretations  of  what  the  enterprise  process  should  be  according  to  individual 
organizations  resulting  in  significant  confusion.  Not  only  does  this  occur  at  AFSPC,  it  is  very 
likely  that  this  occurs  at  SMC,  SAF  and  OSD  and  is  a  problem  throughout  DoD. 

The  following  is  a  summary  of  the  changes  in  the  original  process  outlined  in  the 
published  instructions  as  explained  by  a  source  at  AFSPC.  AFSPC/A8X  is  responsible  for  the 
IPP.  The  IPP  no  longer  produces  a  SMP  (as  the  instruction  indicates).  In  2008,  the  IPP  posted  a 
classified  Space  Re-capitalization  Plan  (SRP),  replacing  the  SMP.  Recently,  the  SRP  was 
replaced  with  the  Service  Core  Functions  (SCF).  The  Air  Force  SCF  Master  Plan  2010  is  the 
most  current  version;  it  is  Air  Force  wide  and  not  specific  to  space.  AFSPC/A8X  is  currently 
working  on  the  AFSCP  Core  Function  Master  Plan  for  both  space  and  cyberspace.  Additionally, 
AFSPC/CC  signed  the  classified  2011  Space  S&T  Guidance  document. 

Bottom  line,  the  formal  process  is  for  AFSPC  to  provide  prioritized  guidance  through  the 
IPP  to  the  S&T  community  on  what  capability  gaps  exist  and  possibly  require  additional 
investment  in  maturing  technologies  to  achieve  those  capabilities.  The  S&T  community  includes 
defense  laboratories,  academia  as  well  as  industry  partners.  Theoretically,  this  guidance 
encourages  them  to  invest  in  technologies  and  concepts  that  could  potentially  become  MDAPs 
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delivering  advanced  capabilities  (this  research  effort  did  not  substantiate  this).  Figure  23 
summarizes  the  flow  for  S&T  efforts  to  meet  MAJCOM  and  COCOM  capability  needs. 


Figure  23:  Science  &  Technology  Process  (Walker/Pawlikowski  2011) 

Now  that  the  basic  S&T  framework  as  defined  by  policy  and  instructions  for  formal 

capability  analysis  and  technology  development  has  been  presented,  it’s  appropriate  to  discuss 
the  informal  and  undocumented  processes.  Interviews  with  SMC,  AFSPC  and  informal 
discussions  with  defense  industry  members  revealed  that  not  all  capabilities  and  concepts  are 
“pulled”  by  AFSPC  through  the  IPP.  Rather,  there  is  a  more  innovative  method  in  which  our 
industry  partners,  laboratories  and  academia  “market”  new  concepts  to  DoD  members.  This  can 
be  as  informal  as  a  discussion  during  a  game  of  golf  about  a  cool  and  innovative  idea  to  the 
actual  demonstration  of  technology  that  delivers  capabilities  not  considered  by  AFSPC. 

ATD,  Paper  Concepts,  Prototypes,  Joint  Capability  Technology  Demonstrations  (JCTD) 
and  Joint  Improvised  Explosive  Device  (IED)  Defeat  Initiatives  are  not  all  inclusive  of  the 
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“capability  development  ideas”  that  the  IPP  is  exposed  to,  but,  represents  a  subset  of  what  likely 
occurs.  The  official  activity  flow  indicates  that  the  IPP  and  the  Applied  Technology  Council 
(ATC)  determine  what  capabilities  to  pursue  further.  The  quickest,  and  often  time  easiest,  path 
is  to  find  an  existing,  approved  ICD  that  the  capability  would  fall  under  (typically  evolutionary). 
If  there  isn’t  an  existing  ICD,  then  depending  on  how  the  proposed  capability  was  injected  into 
the  IPP  will  determine  the  ERAM  path  that  will  result  in  a  JROC  approved  ICD.  The  next 
section  of  this  report  will  discuss  the  initial  stages  of  ERAM  as  depicted  in  Figure  24  and  how  it 
tries  to  balance  the  “as  is”  of  the  process  with  the  official  process  documentation. 
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Initial  ERAM  Activities 


Figure  24:  ERAM  ICD  Check 

Once  a  capability  gap  has  been  identified  or  emerging/maturing  technology  has  been 
successfully  marketed  to  the  DoD,  the  next  phase  begins  with  the  JCIDS  process.  This  can  be 
initiated  one  of  several  ways:  direction/request  for  a  CBA  by  the  responsible  MAJCOM,  a  JCTD, 
a  Prototype,  a  Joint  IED  Defeat  Initiative  (not  discussed — see  Manual  for  Operation  of  JCIDS 
for  further  information),  a  JUON  or  through  the  Advanced  Technology  Demonstrations  (ATD). 

At  AFSPC,  the  IPP  and  ATC  prioritize  the  capability  gaps  and  based  upon  resource 
limitations  (funding,  manpower,  etc.),  determine  a  cutline  with  regards  to  which  will  be  pursued 
further.  For  a  given  proposed  capability  above  the  cutline,  the  first  step  is  to  determine  if  there  is 
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an  approved  ICD  already  exists.  If  there  is,  this  significantly  reduces  the  amount  of  time  to  reach 
MDD.  Without  an  ICD,  JCIDS  heavily  influences  subsequent  activities. 

If  there  is  not  an  existing  ICD,  the  type  of  capability  and  available  technology  will 
determine  what  JCIDS  activity  is  accomplished.  The  CBA  is  the  default  activity.  However,  a 
Military  Utility  Assessment  (MU A)  is  an  alternative  that  can  be  accomplished  for  a  prototype  or 
JCTD.  The  Joint  IED  Defeat  Initiative  Transition  Packet  and  JUONs  are  also  approved 
alternatives  to  the  CBA. 

From  the  Manual  for  Operations  of  JCIDS,  the  analytical  capability  analysis  is 
accomplished  as  a  CBA.  The  purpose  of  a  CBA  is  to  “identify  capability  needs  and  gaps  and 
recommends  non-materiel  or  materiel  approaches  to  address  gaps.”  Additionally,  “It  [the  CBA] 
becomes  the  basis  for  validating  capability  needs  and  results  in  the  potential  development  and 
deployment  of  new  or  improved  capabilities.”  Upon  CBA  completion,  the  report  includes:  1)  a 
description  of  the  mission  and  military  problem  being  assessed,  2)  identification  of  the  tasks  to 
be  completed  to  meet  the  mission  objectives,  3)  identification  of  the  capabilities  required,  4)  an 
assessment  of  how  well  the  current  or  programmed  force  meets  the  capability  needs,  5)  an 
assessment  of  operational  risks  where  capability  gaps  exist,  6)  recommendation  for  possible  non¬ 
materiel  solutions  to  the  capability  gaps,  and  7)  recommendations  for  potential  materiel 
approaches  (if  required).  Should  a  materiel  approach  be  required,  either  AFSPC  or  a  COCOM 
(for  space,  USTRATCOM)  must  sponsor  the  ICD  to  continue  pursuing  the  capability 
development. 

For  prototypes  and  JCTDs,  the  MUA  is  accomplished  at  the  completion  of  the 
demonstration  and  must  have  sufficient  analytical  data  to  be  a  suitable  CBA  replacement.  It  will 
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be  the  basis  for  the  ICD  preparation  and  be  a  key  reference  document  for  the  CDD  and 
Capability  Production  Document  (CPD). 

A  Joint  IED  Defeat  Initiative  Transition  Packet  is  similar  to  the  MUA  in  that  it  must 
provide  sufficient  analytic  analysis  to  support  the  CDD  and  CPD.  This  is  designed  to  streamline 
the  development  and  fielding  of  the  technology  in  support  of  current  combat  operations  within 
US  Central  Command. 

Another  substitute  for  a  CBA  is  when  a  COCOM  submits  a  JUON.  A  JUON  is  defined 
as  an  urgent  operational  need  identified  by  a  combatant  commander  involved  in  an  ongoing 
named  operation.  From  JCIDS,  “The  scope  of  the  combatant  commander  JUON  will  be  limited 
to  addressing  urgent  operational  needs  that:  (1)  fall  outside  of  the  established  Service  processes; 
and  (2)  most  importantly,  if  not  addressed  immediately,  will  seriously  endanger  personnel  or 
pose  a  major  threat  to  ongoing  operations”  (CJCS  201 1).  The  activity  flow  for  JUONs  will  be 
discussed  separately  at  the  end  of  this  section. 

Not  mentioned  in  JCIDS  are  the  S&T  activities  that  may  result  in  a  desired  operational 
capability  after  the  system  has  been  developed  and  launched.  A  recent  example  of  this  occurred 
with  Tactical  Satellite-3  when  it  transitioned  from  an  experimental  demonstration  to  an 
operational  asset  (Air  Force  Magazine  2010).  To  account  for  these  activities,  ERAM 
incorporated  an  Advanced  Technology  Demonstration  as  a  “user  input”  option.  Similar  to  a 
prototype  or  JCTD,  there  will  likely  be  a  MUA  or  CBA  equivalent  report  accomplished  that 
documents  the  analytical  utility  of  the  technology  and  can  be  used  to  support  the  ICD,  CDD  and 
CPD,  if  required. 
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ICD  Development  &  Approval 

Upon  completion  of  the  CBA,  MUA  or  ATD,  the  results  will  be  documented  and  feed 
directly  into  the  ICD,  a  doctrine,  organization,  training,  materiel,  leadership  and  education, 
personnel,  and  facilities  (DOTMLPF)  change  recommendation  (DCR)  or  both.  If  it  is 
determined  a  materiel  solution  is  not  required  and  the  capability  can  be  met  with  existing 
capabilities,  a  DCR  is  written  and  ERAM  terminates.  However,  if  the  CBA  analysis  results  in 


Figure  25:  DOTMLPF  Analysis 

The  first  decision  point  after  it  has  been  determined  an  ICD  is  required  is  to  determine 
who  the  sponsor  is.  For  space,  either  USTRATCOM  (COCOM)  or  AFSPC  (MAJCOM) 
sponsors  the  ICD.  If  it  is  sponsored  by  the  COCOM,  this  eliminates  several  Air  Force  specific 
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requirements,  as  shown  in  the  activity  diagram.  However,  if  the  ICD  is  sponsored  by  the 
MAJCOM,  the  next  activity  is  the  RSR.  The  purpose  of  this  review  is  for  the  sponsor  (AFSPC) 
to  identify  the  funding  strategy  for  the  Materiel  Solution  Analysis  and  Technology  Development 
phases.  As  stated  in  AFI  63-101,  prior  to  milestone  approval,  funding  must  be  available  and 
identified  to  reach  the  next  milestone.  In  this  case,  funding  must  be  available  from  JROC 
approval  of  the  ICD  through  MS-A  (SAF/AQ  2009). 

Upon  approval  at  the  RSR,  AF/A5RD  establishes  and  facilitates  a  HPT  with  the  objective 
of  writing  the  ICD.  The  HPT  is  led  by  the  sponsor  (AFSPC)  and  ideally  has  7-11  core  members 
consisting  of  space  SMEs  from  AFSPC,  SMC,  FFRDC  and  government  support  contractors. 
“This  core  team’s  objective  is  to  capture,  articulate  and  document  the  operator’s  requirements  in 
minimum  time,  while  achieving  stakeholder  buy-in”  (AFSPC/A3F  2007).  One  advantage  of  the 
HPT  is  it  facilitates  simultaneous  coordination  with  Air  Force,  Joint  Staff,  Service  and  Agency 
coordination.  Failure  to  utilize  the  HPT  results  in  sequential  staffing  of  the  document. 

Once  the  ICD  has  been  drafted  by  the  HPT,  it  enters  0-6  level  coordination.  For  Air 
Force  coordination,  this  is  accomplished  with  the  Information  and  Resource  Support  System 
(IRSS).  The  Knowledge  Management/Decision  Support  (KM/DS)  database  is  used  for  joint  0-6 
and  flag  officer  coordination  of  JCIDS  documents.  This  activity  in  the  framework  includes  the 
time  required  to  submit  the  document  for  coordination  as  well  as  time  for  review  and  comment 
resolution. 

After  document  coordination  and  comment  resolution  is  complete,  the  ICD  is  ready  to  be 
reviewed  by  the  AFROC.  This  activity  includes  the  lead  time  required  to  get  on  the  AFROC 
agenda  and  complete  the  formal  review.  If  approved  by  the  AFROC,  the  document  will  proceed 
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to  the  Gatekeeper,  the  J-8/Deputy  Director  for  Requirements,  for  review  and  document 
designation. 

The  Gatekeeper  will  review  all  JCIDS  documents  regardless  of  ACAT,  previous 
delegation  decisions,  or  previous  joint  potential  designator  (JPD)  decisions  and  assign  a  JPD. 
Possible  JPDs  are  “JROC  Interest”,  “Joint  Capabilities  Board  (JCB)  Interest”,  “Joint 
Integration”,  “Joint  Information”  or  “Independent”.  Of  the  nine  Joint  Capability  Areas  (JCA), 
four  of  the  areas  automatically  bin  the  capability  as  “JROC  Interest”;  from  AFSPC  sources, 
space  almost  always  falls  in  one  of  these  four  JCAs.  Table  7  from  the  Manual  for  Operation  of 
JCIDS  indicates  the  offices  required  to  coordinate  with  depending  on  JPD  assignment. 


Table  7:  Joint  Staffing  Matrix  (CJCS  2011) 


Table  D-l.  Staffing  Matrix 


Office 

JROC 

Interest 

JCB 

Interest 

Joint 

Integration 

Joint 

Information 

Indepen¬ 

dent 

Army 

X 

X 

X 

X 

S 

Navy 

X 

X 

X 

X 

S 

Air  Force 

x 

X 

X 

X 

s 

Marine  Corps 

X 

X 

X 

X 

s 

Joint  Staff 

X/C/E 

X/C/E 

C/E 

X 

FCB  Working 
Groups 

L/S 

L/S 

L/S 

L/S 

L/S 

Combatant 

Commanders 

X 

X 

X 

X 

s 

Other  DOD 
Components 

X 

X 

X 

X 

s 

USDIAT&LI 
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X 

X 

X 

USDIP) 

X 

X 

X 

X 

USDII) 

X 

X 

X 

X 

USecAF  (DOD 

EA  for  Space) 

X 

X 

X 

X 

s 

ASDINIII/CIO 

X 

X 

X 

X 

USDIPA.R) 

X 

X 

X 

X 

USD(C) 

X 

X 

X 

X 

DOT&E 

X 

X 

X 

X 

Director. 

CA&PE 

X 

X 

X 

X 

D1A 

X 

X 

X 

X 

DISA 

X 

X 

X 

X 

s 

NGB 

X 

X 

X 

X 

s 

NGA 

X 

X 

X 

X 

s 

NSA 

X 

X 

X 

X 

s 

NRO 

X 

X 

X 

X 

s 

DNI/IRB 

X 

X 

X 

X 

L/S  =  lead /supporting  FCB 
S  =  Sponsor  staffing  only 
X  =  Required  staffing 
C  =  Certification 

E  =  Weapon  Safety  Endorsement 
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Figure  26  summarizes  the  flow  of  activities  and  timelines  for  a  JPD  of  “JROC  Interest” 
or  “JCB  Interest”.  This  is  essentially  the  detailed  version  of  the  circled  ERAM  activities  in 
Figure  25.  However,  in  ERAM  2.1,  the  model  does  not  specifically  identify  each  of  the  activities 
from  Figure  26,  but,  rather  incorporates  all  the  times  into  the  triangular  distribution  for  each  of 
the  major  reviews  (Functional  Capabilities  Board  (FCB),  JCB  and  JROC). 


JROC  Interest 
or  JCB  Interest 


X 


Certification  endorsement 
Reviews  (J-2.J-6.DDFP) 

- H - 


Document  Review 
Phase  1  (21  Days) 


Sponsor 
ADJUDICATE 
COMMENTS 
<45  Djy  SmptJi!*) 


Certification  endorsement 

Document  Review 

Reviews  (J-2.  J-6.  DDFP) 

Phase  2  (21  Days) 

Sponsor 

ADJUDICATE 

COMMENTS 

(15  Dav  Suspcme) 


FCB  Draft 
(15  Days) 


"E 


Final  Certifications 
Endorsements  (7-2, 
J-6,  DDFP) 

T 


FCB 

Review 


D-8 


Filial  Document 
To  Database 


JCB/ JROC 

Validation 

Approval 


Sponsor/ 

PM 


Figure  26:  JROC  Process  (CJCS  2011) 

The  ERAM  framework  simplifies  this  process,  but  does  account  for  all  the  time  required 
to  coordinate  a  document  through  the  Joint  Staff  and  have  it  approved.  Figure  27  was  a  valuable 
source  of  data  for  ERAM  activity  times;  it  specified  both  regulatory  and  most  likely  times  to 
staff  a  JCIDS  “JROC  Interest”  document  through  the  Joint  Staff. 
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JROC  Interest  Document  Timeline  Calculator 

Last  Update:  8  Dec  09 

Backwards  from  "Need"  Date 

Step 

Event 

Regulatory  Goal 

Realistic 

Days 

Date 

Days 

1 

Submit  RSR  Request  to  AF/A5RP 

5-Aug-11 

^16-Apr-11 

A 

Zero  out  days  for"Flag-Level  Review’’ 
and  '’Sponsor  Resolves  Flag-Level 
Comments”  if  you  anticipate  resolving 
all  Critical  comments 

2 

RSR  Briefing  to  AF/A5R 

21 

26-Auq-11 

21 

^T*Moy  1 1-** 

3 

HPT  Commences 

30 

25-Sep-11 

30 

6-Jun-ll 

4 

HPT  Ends 

11 

6-Oct-11 

11 

17-Jun-ll 

5 

Internal  Coordination  and  Cleanup 

30 

5-Nov-ll 

60 

16-Aug  11 

6 

0*6  Review  (MAJCOM.  Air  Staff  Joint  Staff) 

21 

26-Nov-ll 

28 

13-Sep-11 

7 

Sponsor  Completes  0-6  Level  Comment  Resolution 

45 

IO-Jan-12 

60 

12JJotTT 

8 

Submit  CRM/Doc/Brief  for  AFROC 

14 

24-Jan-12 

26-Nov-l  1 

9 

AFROC 

16 

9-Eeb-^T' 

16 

12-Dec- 11 

10 

Flag-Level  Review  (if  required) 

214^ 

^1-Mar-12 

28 

9-Jan-12 

11 

Flaq-Level  Comment  Resolution  (if  required) 

15* 

16-Mar-12 

30 

8-Feb-12 

12 

FCB 

21 

6-Apr-12 

28 

7-Mar  12 

13 

JCB 

14 

20-Apr-12 

21 

28-Mar-12 

14 

JROC 

7 

27-Apr-12 

14 

11 -Apr-1 2 

15 

Document  Siqned 

14 

11 -May-1 2 

30 

11-May-12 

TOTAL  #  of  Days 

280 

391 

Enter  the  date  of  your  Milestone  Decision  in 
the  green  highlighted  cell  (D-17). 

“DO  NOT  PLAN  TO  THIS  TIMELINE** 

Date  when  you  ’’realistically" 
need  to  submit  your  RSR 
request  by. 

Figure  27:  JROC  Document  Timeline  Calculator  (JROC  2009) 

Post  ICD  -  Pre  MDD 

Once  the  ICD  has  been  approved  by  the  JROC,  this  signifies  the  Joint  Staffs  validation 
of  the  need  to  fill  a  capability  gap  with  a  material  solution.  In  the  ERAM  framework,  the  next 
formal  activity  is  DP,  as  depicted  in  Figure  28. 
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Figure  28:  Early  SE  Design 

According  to  AFI  63-101,  “DP  is  the  materiel,  i.e.,  implementing  command’s  (AFMC 
and/or  AFSPC),  contribution  to  AF  or  AF-led  capability  planning.  It  is  a  collaborative  process 
bridging  warfighter-identified  capability  needs  to  planning  for  acquisition  of  materiel  solutions.” 
Within  ERAM,  this  activity  begins  following  the  approval  of  the  ICD,  however,  in  reality,  DP 
can  start  as  early  as  when  the  HPT  is  formed  to  draft  the  ICD;  this  helps  jump  start  the  activity 
and  reduce  overall  time.  AFSPC/ A5  would  begin  organizing  the  DP  team  while  the  HPT  drafts 
the  ICD  and  it  works  through  the  Joint  Staff  coordination  process  and  have  begun  coordinating 
with  the  appropriate  personnel  at  SMC  to  get  them  up  to  speed.  Additionally,  in  ERAM,  DP 
activities  at  the  Center  are  those  required  to  generate  the  CCTD(s).  In  reality,  this  effort  may 
continue  through  the  Materiel  Solution  Analysis  phase  of  the  acquisition  system,  depending  on 
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the  requirements  of  the  capability  and  associated  concept(s).  Figure  29  from  the  DP  Guide 
illustrates  when  this  activity  occurs  while  Figure  30  describes  the  detailed  activities. 


Figure  29:  DP/JCIDS  Process  (AFMC/A2/5  2010) 


INPUTS 
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Others 
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Figure  30:  Early  SE  Process  (SAF/AQ  2010) 
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According  to  the  AFMC  DP  Guide,  DP  activities  are  led  by  AFSPC  and/or  either 
SMC/XR  (revolutionary  capability)  or  the  existing  SPO  XR  shops  (evolutionary  capability). 

They  are  responsible  for  providing  technical  support  at  the  RSR,  AFROC,  JROC,  Air  Force 
Review  Board  (AFRB),  MDD  and  AoA.  The  products  that  are  being  developed  during  this 
activity  include  the  CCTDs,  draft  AoA  Study  Guidance  and  the  AoA  Study  Plan.  Policy  requires 
the  AoA  Study  Guide  and  Study  Plan  to  be  completed  prior  to  MDD. 

Based  on  discussions  with  AFSPC  and  to  simplify  the  modeling  of  this  process,  in 
ERAM,  the  official  DP  request  occurs  following  the  approval  of  the  ICD.  This  request  is  sent  to 
either  SMC/XR  or  to  the  SPO,  depending  on  if  the  capability  is  revolutionary  or  evolutionary, 
respectively.  SMC/XR  or  the  SPO  develops  a  proposal  including  number  of  resources  and 
funding  requirements  for  the  effort  and  submits  this  to  AFSPC  for  review.  At  this  point,  AFSPC 
will  either  approve  the  proposal  as  is  or  kick  it  back  to  SMC  for  re-work.  If  it  is  approved, 
AFSPC  verifies  funding  is  available;  if  it  isn’t,  they  look  for  alternate  funds.  If  alternate  funds 
are  not  available,  the  effort  is  shelved.  If  funds  are  found,  then  the  DP  activities  kick  off. 

AFSPC  and  SMC  work  together  to  develop  the  CCTDs,  the  AoA  Study  Guide  and  the  AoA 
Study  Plan.  In  order  to  proceed  with  drafting  the  ADM  and  meeting  the  AFRB,  at  a  minimum, 
the  AoA  Study  Guide  and  Plan  must  be  completed.  The  CCTDs  necessary  for  the  AoA  effort 
ideally  would  be  completed  as  part  of  the  DP  activity,  but,  will  not  hold  up  the  MDD. 

According  to  AFSPC,  there  are  very  few  official  DP  requests  for  space  systems  and  all  requests 
are  approved  and  funded. 
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MDD 

Upon  completion  of  the  DP,  preparation  activities  for  the  MDD  begin.  The  first  activity 
following  DP  completion  is  the  AFRB.  According  to  AFI  63-101,  paragraph  3. 7. 1.1,  AFRBs  are 
not  conducted  for  services  or  space  programs.  After  inquiring  with  SAF/AQR  about  this 
statement,  clarification  was  given  that  Mr.  Van  Buren,  SAF/AQ,  has  implemented  the 
requirement  for  space  programs  to  participate  in  the  AFRB  in  advance  of  the  MDD.  This  is  to 
ensure  the  SAE  has  been  coordinated  with  and  is  prepared  to  fully  endorse  the  MDD  to  the 
MDA.  ERAM  has  included  this  activity  within  the  framework  and  model.  The  AFRB  activity 
includes  the  time  required  to  draft  and  coordinate  the  MDD  briefing  and  ADM.  From  DoDI 
5000.02,  the  purpose  of  the  MDD  is: 

When  the  ICD  demonstrates  the  need  for  a  materiel  solution,  the  JROC  shall  recommend 
that  the  MDA  consider  potential  materiel  solutions.  The  cognizant  MDA  is  determined  as 
described  in  Enclosure  3.  The  MDA,  working  with  appropriate  stakeholders,  shall 
determine  whether  there  is  sufficient  information  to  proceed  with  a  Materiel 
Development  Decision.  If  the  MDA  decides  that  additional  analysis  is  required,  a 
designated  office  shall  prepare,  and  the  MDA  shall  approve,  study  guidance  to  ensure 
that  necessary  information  is  available  to  support  the  decision 
(OUSD/ATL  2008). 

For  space  systems,  the  MDA  is  the  Under  Secretary  of  Defense,  AT&L  (USD/AT&L).  Prior  to 
meeting  with  the  MDA,  preparation  activities  are  jointly  shared  by  AFSPC  and  SMC.  From  the 
Pre-MDD  Guidebook,  the  briefing  team  will  present  the  JROC  recommendation  with  regards  to 
the  ICD,  and  justify  the  business  case  identifying  the  affordability,  viability  of  the  materiel 
solution  to  address  the  gap(s).  Additionally,  they  must  justify  why  an  Analysis  of  Alternatives  is 
not  required  (if  it  is  proposed  not  to  do  one)  or  present  the  AoA  Study  Plan  and  Study  Guide. 

At  the  conclusion  of  the  MDD,  the  MDA  will  issue  an  ADM  that  includes  signed  AoA 
Study  Guidance,  authorizes  an  executive  steering  group  to  oversee  the  AoA  (if  required), 
identifies  what  phase  of  the  acquisition  system  the  program  will  enter  (pre  MS-A,  MS-B  or  MS- 
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C),  designates  a  lead  service  component  (typically  Air  Force  for  space)  and  identifies 
expectations  for  the  initial  milestone  review.  The  phase  of  the  acquisition  cycle  that  will  be 
directed  is  technically  reflected  within  the  contents  of  the  CCTD(s)  and  will  be  summarized  in 
the  AoA  Study  Guidance.  These  will  document  the  maturity  level  of  the  technology  required  to 
fill  the  gap  and  recommend  whether  a  materiel  solutions  analysis,  technology  development  or 
engineering  and  manufacturing  phase  is  appropriate  for  entry.  It  is  possible  for  the  ADM  to 
approve  entry  into  a  later  phase  and  give  specific  direction  on  additional  requirements  necessary 
(AFMC/OAS  2010). 

Post  MDD 

Following  the  signing  of  the  ADM  at  the  MDD,  a  significant  amount  of  logic  was 
inserted  into  the  model  to  determine  the  probabilities  of  where  the  capability  would  like  be 
inserted  into  the  acquisition  system.  The  first  attribute  considered  is  whether  the  capability  is 
evolutionary  (upgrading  an  existing  system  in  and  existing  SPO)  or  revolutionary  (new  system 
and  a  new  SPO).  If  the  system  is  revolutionary,  it  is  assumed  that  an  AoA  must  be 
accomplished.  Depending  on  what  type  of  system  is  being  considered  (JCTD,  Prototype,  ATD 
or  Paper  Concept)  has  unique  probabilities  for  placement  in  the  acquisition  system.  Since  a 
JCTD,  prototype  and  ATD  are  proven  prototypes,  it  is  much  more  likely  that  they  will  enter  the 
system  pre  MS-B  or  pre  MS-C.  If  the  revolutionary  system  is  a  paper  concept,  following  the 
AoA  it  is  highly  likely  there  will  be  technology  development  required  and  it  will  enter  the  system 
pre  MS-A.  If  the  system  is  evolutionary,  the  likelihood  that  a  previously  accomplished  AoA 
may  be  used  is  high.  Depending  on  if  the  system  is  a  JCTD,  prototype,  ATD  or  paper  concept 
will  impact  the  probabilities  of  what  phase  of  the  acquisition  system  it  will  enter.  For  example,  a 
paper  concept  will  more  than  likely  enter  pre  MS-A  since  technology  development  is  likely 
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required  whereas  a  JCTD  will  likely  be  able  to  enter  at  pre  MS-C  since  it  is  an  upgrade  to  an 
existing  system  and  the  technology  has  been  demonstrated.  Table  8  summarizes  the  logic  used 
within  ERAM  2. 1  for  placement  in  the  acquisition  system  following  the  MDD  and  AoA  activity, 
if  required. 


Table  8:  Acquisition  System  Insertion  Logic 


REVOLUTIONARY  EVOLUTIONARY 


*  AoA  is  always  accomplished 

|  AoA  Exists:  80% 

AoA  Required:  20% 

Pre  MS-A 

Pre  MS-B 

Pre  MS-C 

1  Pre  MS-A 

Pre  MS-B 

Pre  MS-C 

Pre  MS-A 

Pre  MS-B 

Pre  MS-C 

5% 

75% 

20% 

JCTD 

5% 

20% 

75% 

5% 

75% 

20% 

45% 

50% 

5% 

Prototype 

15% 

75% 

10% 

45% 

50% 

5% 

45% 

50% 

5% 

ATD 

15% 

75% 

10% 

45% 

50% 

5% 

100% 

0% 

0% 

Paper  Concept 

95% 

5% 

0% 

100% 

0% 

0% 

With  this  logic  established,  ERAM  1.0  sufficiently  identified  the  activities  following  the 
AoA  and  no  additional  framework  was  developed  for  this  research  effort.  To  validate  the 
accuracy  of  the  flow  of  the  ERAM  2. 1  framework,  it  was  shared  with  several  SMEs  at  the  Air 
Staff,  AFSPC  and  SMC.  After  approximately  1 1  iterations,  the  flow  of  the  activity  diagram  was 
generally  agreed  upon.  This  is  due  in  large  part  to  the  various  interpretations  that  each  SME  had 
with  regard  to  the  “spirit  and  intent”  of  the  process.  The  next  portion  of  this  phase  was 
collecting  the  data  for  each  activity  and  decision  point. 

ERAM  uses  triangular  distributions  of  time  for  the  activities  while  the  decision  points 
require  percentages  of  occurrence  for  each  outcome,  the  sum  being  100%.  This  was  both  the 
most  challenging  portion  of  the  research  as  well  as  the  most  critical,  since  it  is  the  basis  of 
ERAM  outputs.  There  were  two  data  sources  available:  1)  written  documentation  (policy, 
instructions  and  guides)  and,  2)  SMEs.  The  problem  with  the  written  documentation  is  too  often, 
it  didn’t  represent  reality.  There  are  nuances  in  the  activities  not  accounted  for  in  the  policy 
timelines  such  as  sufficient  time  for  coordination  and  comment  resolution.  These  nuances  were 
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captured  during  the  interviews  with  the  SMEs  and  explain  the  difference  between  policy  and 
reality.  A  challenge  to  overcome  with  the  SMEs  was  there  were  few  that  had  been  in  positions 
for  a  significant  period  of  time  to  be  able  to  keep  track  of  this  type  of  information  and  answer 
specific  questions.  The  processes  and  overarching  guidance  that  currently  exist  have  frequently 
changed  in  the  past  decade,  making  it  difficult  to  answer  questions  on  the  process  as  it  is 
modeled  in  ERAM.  Additionally,  JCIDS  activities  for  space  systems  do  not  occur  frequently 
enough  to  establish  a  well  defined  data  set.  Therefore,  as  required  with  other  complex  discrete 
modeling  simulations,  the  research  team  leveraged  heuristics  of  the  SMEs  and  encouraged  them 
to  estimate  times  based  on  their  expert  opinions. 

To  begin  data  collection,  two  tables  were  built  that  followed  the  traditional  space 
capability  development  path  (not  the  JUON  path).  Table  9  included  all  the  activities  created  in 
the  ERAM  2.1  framework,  including  available  published  times  (from  policy  and  instructions). 
Additional  columns  were  added  to  identify  the  least  number  of  days  required  (best  case),  the 
most  likely  days  required  and  the  most  days  required  (worst  case).  Table  10  identified  all 
decision  points  in  the  framework,  the  possible  outcomes  and,  based  on  heuristics  and  historical 
data,  what  percentage  of  time  each  outcome  occurred.  For  decision  analysis,  policy  and 
instructions  do  not  provide  this  type  of  data — this  was  collected  solely  from  SMEs.  Both  tables 
were  shared  with  SMEs  at  the  Air  Staff,  AFSPC  and  SMC  with  the  request  that  they  share  their 
personal  opinions  on  the  times,  all  of  which  was  based  on  historical  data  and  experience  with 
current  processes. 
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Table  9:  Pre  MS-A  ERAM  Activity  Timelines 


ACTIVITY 

PUBLISHED 
TIME  (days) 

LEAST  (days) 

MOST 

LIKELY  (days) 

MOST  (days) 

CBA 

90-180 

(JCIDS) 

90  (evolutionary) 
180  (revolutionary) 

150 

(evolutionary) 

540 

(revolutionary) 

180 

(evolutionary) 

720 

(revolutionary) 

MUA 

120 

180 

360 

JUON 

90 

180 

300 

RSR 

30 

60 

90 

HPT 

71 

71 

101 

131 

0-6  Coord 

80 

80 

102 

123 

AFROC 

52 

14 

74 

120 

Draft  ICD 

71 

101 

131 

Joint  Coord 

80 

102 

123 

FCB 

21 

21 

28 

42 

JCB 

14 

14 

21 

35 

JROC 

21 

21 

44 

65 

DP  Request 

3 

5 

10 

DP  Proposal 

30 

45 

90 

Proposal  Review 

3 

7 

21 

DP  Rework 

20 

30 

40 

Find  Alternate 
Funds 

10 

20 

30 

DP 

240 

360 

720 

Candidate 
Solution  Sets 
Selection 

Initial  Concepts 
Review 

Concept 

Characterization 

Review 

Final  Concepts 
Review 

Release 

Approval 

CCTD  Development  Concurrent  with  DP  Activity 

AFRB 

14 

21 

42 

MDD 

14 

42 

84 

MDD  Rework 

30 

90 

180 

AoA 

180 

360 

720 

The  cells  that  are  not  shaded  indicate  data  was  provided  from  the  SMEs  based  on  their 


expert  opinion  for  an  AC  AT  ID  space  program.  For  those  cells  shaded  in  gray,  the  SMEs  were 


unable  to  provide  discrete  answers;  based  on  their  own  expert  opinions,  the  authors  estimated  the 
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time  required  for  these  activities.  The  following  describes  the  activities  in  the  ERAM  2.1 
framework  and  the  work  associated  with  each  activity: 

CBA:  AFSPC  responsibility  that  requires  90-180  days  for  evolutionary  systems  and  180-720 
days  for  revolutionary  systems.  Evolutionary  systems  are  those  that  are  upgrades  to  existing 
systems  and  have  an  established  SPO;  revolutionary  systems  are  new  systems  and  will  require  a 
new  SPO. 

MUA:  120-360  days  to  write  an  MUA  and  have  it  approved 

JUON:  the  initial  JIJON  framework  was  identified,  however,  event  distributions  were  not 
obtained  and  require  additional  research  efforts.  Rather  than  discard  the  framework,  it  was 
assumed  in  the  model  coding  that  the  comprehensive  distribution  of  time  for  responding  to  a 
JUON  is  at  least  90  days,  most  likely  180  days  and  at  most,  300  days. 

RSR:  for  AFSPC  sponsored  ICDs,  AFSPC/A5X  develops  the  RSR  briefing  (30-90  days), 
submits  request  to  AF/A5RP  for  RSR  21  days  (policy  requirement)  in  advance  of  brief  (JCS 
2009) 

HPT:  the  HPT  takes  30  days  (both  regulatory  goal  and  realistic)  to  establish  the  team  and 
schedule  the  activity  following  RSR  approval.  Per  policy,  the  HPT  itself  takes  1 1  days  and 
another  30  days  are  scheduled  for  internal  coordination  and  cleanup  (60  days  realistic).  This 
results  in  71  days  total  (policy)  or  101  days  (realistic)  (JCS  2009). 

0-6  Review:  this  activity  includes  both  the  0-6  review  at  the  MAJCOM,  Air  Staff  and  Joint 
Staff  as  well  as  the  time  required  to  submit  the  comment  resolution  matrix  (CRM),  document  and 
briefing.  The  expected  time  is  21  days  for  document  review  (policy),  28  days  (realistic);  45  days 
for  comment  resolution  (policy),  60  days  (realistic);  14  days  for  submitting  CRM,  document  and 
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brief  (policy  and  realistic).  Therefore,  policy  goal  total  is  80  while  realistically,  it  is  102  days 
(JCS  2009). 

AFROC:  this  activity  includes  both  the  AFROC  meeting  and,  if  required,  flag  officer  review  and 
comment  resolution.  The  AFROC  requires  16  days  to  review  documents  (policy  and  realistic) 
and  approve/disapprove.  Should  a  critical  comment  be  unresolved  below  the  flag  officer  level, 
the  flag  officer  review  requires  an  additional  21  days  (policy),  28  days  (realistic).  Flag  officer 
comment  resolution  requires  an  additional  15  days  (policy),  30  days  (realistic).  SAF/AQ  sources 
indicated  that  rarely  are  flag  officer  review  and  comment  resolution  required,  therefore,  the 
AFROC  activity  in  ERAM,  the  least  number  of  days  is  16,  the  most  likely  number  of  days 
requiring  flag  officer  review  and  taking  74  days  (realistic)  and  the  maximum  number  of  days  is 
120  (JCS  2009). 

Draft  ICD:  similar  to  the  HPT  activity  that  occurs  for  Air  Force  sponsored  ICDs,  this  is  the 
activity  that  occurs  when  the  ICD  is  sponsored  by  the  COCOM.  It  incorporates  time  to  establish 
the  team,  schedule  the  meeting  and  hold  the  meeting  where  the  ICD  will  be  drafted.  Despite  the 
fact  that  the  COCOM  sponsors  this,  much  of  the  work  is  accomplished  at  AFSPC.  Therefore, 
the  same  distributions  from  the  HPT  were  used  for  this  activity. 

Joint  Coord:  similar  to  the  0-6  Coord  activity  for  Air  Force  sponsored  ICDs,  this  is  the 
COCOM  sponsored  activity  for  coordinating  the  drafted  ICD.  The  distributions  for  0-6  Coord 
were  used  for  this  activity  since  they  accomplish  the  same  purpose. 

FCB:  this  activity  occurs  21  days  (policy)  after  the  AFROC  activities,  28  days  (realistic)  (JCS 
2009). 

JCB:  this  activity  occurs  14  days  (policy)  after  the  FCB,  21  days  (realistic) 
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JROC:  this  activity  occurs  7  days  (policy)  after  the  JCB,  14  days  (realistic);  14  days  (policy)  to 
sign  the  document,  30  days  (realistic)  for  a  total  of  21  days  (policy)  or  44  days  (realistic). 

DP  Request:  3-10  days  for  AFSPC/A5  to  draft  a  request  for  DP  support  tasked  to  SMC/XR  or 
SPO  XR  (revolutionary  or  evolutionary  capability,  respectively). 

DP  Proposal:  SMC/XR  or  SPO  XR  has  30  days  to  submit  DP  proposal  to  AFSPC,  however, 
based  on  SME  experience,  this  can  take  up  to  90  days  for  the  Center  to  accomplish. 

Proposal  Review:  after  the  proposal  is  completed  by  SMC/XR  or  the  SPO/XR,  AFSPC  reviews 
the  proposal. 

DP  Rework:  should  the  proposal  not  be  approved  and  require  changes,  it  is  sent  back  to  the 
originating  XR  branch  for  rework. 

Find  Alternate  Funds:  should  funds  not  be  immediately  available  for  the  DP  effort,  AFSPC 
will  work  to  find  alternate  funds  for  the  effort.  In  the  unlikely  event  they  are  not  found,  the 
effort  will  terminate. 

CCTD:  this  Center  led  activity  occurs  concurrently  with  the  DP  effort  being  accomplished  by 
AFSPC.  Specific  activities  include  Candidate  Solution  Sets  Selection,  Initial  Concepts  Review, 
Concept  Characterization  Review,  Final  Concepts  Review  and  Release  Approval.  This  will  be 
accomplished  prior  to  the  completion  of  the  DP  activity  and  is  therefore  not  assigned  duration 
distributions. 

DP:  this  activity  concludes  when  AFSPC  receives  the  CCTDs  from  the  Center;  AFSPC  develops 
the  ICD  briefing  including  “Level  of  Investment”,  Affordability  Assessment,  JROC 
Recommendation,  AoA  Study  Guidance  and  AoA  Study  Plan. 

AFRB:  this  activity  includes  the  number  of  days  to  draft  the  MDD  briefing  and  the  ADM  and 
get  SAE  approval  to  proceed  to  the  MDA. 
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MDD:  this  is  the  activity  where  the  MDD  briefing  is  presented  and  signature  for  the  ADM  is 
acquired. 

MDD  Rework:  should  the  MDA  not  approve  the  ADM  and  require  rework,  this  activity 
captures  the  time  required  for  this  effort. 

AoA:  the  AoA  activity  may  take  as  few  as  180  days  or  up  to  720  days  to  complete.  If  CCTDs 
are  done  with  the  appropriate  level  of  technical  analysis,  according  to  DP  SMEs  at  AFSPC  and 
SMC,  this  should  reduce  the  amount  of  time  AoAs  took  in  the  past  (12-24  months)  to  just  6-12 
months,  depending  on  the  complexity  of  the  system  and  available  resources. 


Table  10:  ERAM  Pre  MS-A  Capability  Development  Decision  Probabilities 


Decision 

Outcome  (Probability  %) 

Existing  ICD 

YES  (90) 

NO  (10) 

DOTMLPF 

ICD  and/or 
DCR  (95) 

DCR  (5) 

RSR  Approve 

YES  (50) 

NO  (50) 

AFROC 

YES  (60) 

NO  (40) 

FCB 

YES  (50) 

NO  (50) 

JCB 

YES  (75) 

NO  (25) 

JROC 

YES  (90) 

NO  (10) 

DP  Approve 

YES  (99) 

NO  (1) 

DP  Funding 

YES  (99) 

NO  (1) 

MDD  ADM 

YES  (90) 

NO  (10) 

AoA  Exists 

YES  (50) 

NO  (50) 

AoA  Funding 

YES  (80) 

NO  (20) 

JUON  Post 
Deployment 

JCTD  (90) 

Theater 

Sustainment  (5) 

Demilitarize  (5) 

The  probabilities  in  Table  10  were  the  most  difficult  to  determine.  Unique  personnel 


characteristics  such  as  rank/level  of  capability  sponsorship,  personality  of  SMC  and  AFSPC 
briefing  team  to  gain  HHQ  buy-in,  as  well  as  the  political  environment  all  impact  the 
probabilities  of  the  outcome  for  a  specific  instance.  In  correspondence  with  the  SMEs,  they 
struggled  to  provide  quantitative  data  (probabilities)  to  these  questions.  The  notional 
probabilities  are  based  on  the  expert  opinions  of  the  authors  and  further  sampling  is  required  to 
increase  the  fidelity  of  the  data. 
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Rapid  Capability  Development 

In  addition  to  the  traditional  capability  development  methods  discussed  thus  far,  there 
also  exists  a  more  streamlined  process  for  rapid  capability  development.  CJCSI  3170.01G 
accounts  for  this  process  and  defines  it  as  a  Joint  Urgent  Operational  Need  (JUON). 
Additionally,  AFI  10-614  provides  further  instruction  on  this  process.  Based  on  the  guidance 
documents  as  well  as  a  SME  from  SAF/AQR,  the  initial  framework  for  this  process  was 
modeled,  however,  the  data  for  the  activities  and  decision  points  was  not  acquired.  This  will  be 
included  as  a  recommended  topic  for  future  ERAM  research. 

Table  1 1  is  a  summary  of  the  activities  and  decisions  identified  in  the  rapid  capability 
development  flow  (JUON)  for  ERAM  2. 1 .  Further  research  will  be  required  to  determine  the 
distributions  of  times  required  for  the  activities  as  well  as  the  probabilities  associated  with  the 
decision  points.  The  framework  was  derived  from  CJCSI  3170.01G,  AFI  10-614  and  an 
interview  with  a  SAF/AQR  SME.  Rather  than  eliminate  the  initial  framework,  it  remains  for 
further  research  to  be  accomplished  and  a  single  distribution  was  assumed  for  all  activities 
inclusively  (see  Table  9).  Following  the  delivery  of  a  capability  in  response  to  a  JUON,  three 
options  are  available  and  are  denoted  in  the  ERAM  2. 1  framework  as  a  CTR  Decision.  The 
system  can  be  demilitarized  and  disposed  of  in  theater,  it  can  be  sustained  in  theater  and  funded 
by  the  COCOM,  or  it  may  re-enter  the  JCIDS  process  as  a  JCTD.  Table  10  indicates  the 
probabilities  the  authors  assumed  for  this  decision  point;  further  research  is  required  to  improve 
this  portion  of  the  model. 
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Table  11:  JUON  Activities  and  Decisions  for  Future  Research 


ACTIVITY 

DECISION 

J8  Review 

J8  Approve 

JRAC 

Sponsor 

Funding 

Space 

AFSPC  Tasking 

COA 

QRC-IPT  COA 

MDD  Approve 

QRC  MDD 

CTR  Decision 

MDD  ADM 

Development 

Test  &  Field 

4.4  Phase  4:  Implement  design  updates  to  ERAM  1.1,  1.2,  2.0  &  2.1 
ERAM  1.1 

ERAM  1 . 1  was  implemented  by  the  Aerospace  design  team  to  make  the  required  updates 
due  to  the  requirements  of  the  new  simulation  software.  This  effort  was  accomplished  by  the 
programmers  in  discussions  with  Wirthlin,  the  software  modeling  programmer  for  ERAM  1 .0. 
These  minor  changes  served  as  the  ExtendSim®  baseline  for  further,  more  significant  model 
updates. 

ERAM  1.2 


Table  12:  DoDI  5000.02  Model  Change  Summary 


Model  Change 

i 

Move  PDR  to  Pre-B,  Add  SDR  (Uncertainty  Flow) 

2 

Move  PDR  to  Pre-B,  Add  SDR  (Main  Flow) 

3 

SDR  Submodel 

4 

Space  Specific  Submodel 

5 

“What  If’  Submodel 

6 

Cost  Growth  Check 

Table  12  summarizes  the  changes  to  ERAM  1.0  that  resulted  from  applying  updated 
space  policy  guidance.  Table  12  describes  the  data  entered  into  each  activity  model  for  the 
triangular  distributions  of  the  timeframes.  Table  13  describes  the  probabilities  for  the  decision 
points  in  the  model  for  DoDI  5000.02  and  space  policy  updates. 
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Table  13:  DoDI  5000.02  Update  Timelines 


ACTIVITY 

PUBLISHED 
TIME  (days) 

MINIMUM 

(days) 

MOST 

LIKELY 

(days) 

MAXIMUM 

(days) 

Responsible 

Organization 

Clinger-Cohen  Act 
Compliance/ Assess 
ment 

180 

180 

180 

210 

SPO 

Rework  from  MDA 

60 

90 

180 

SPO 

IP  A  Calendar  Time 

7 

10 

30 

SPO 

Cost  Growth  Check 

14 

30 

60 

SPO 

Significant  Change 
Notice 

45 

30 

45 

60 

SPO 

MDA  Approval 
Process  (Cost 
Growth) 

60 

70 

90 

MDA 

Table  14:  DoDI  5000.02  Decision  Probabilities 


Decision 

Probability 

IPA  Required 

YES  (65%) 

NO  (35%) 

MDA  Approval 
(Post  Design 
Review) 

YES  (65%) 

NO/Rework  (30%) 

NO/Kill  (5%) 

SDR  Success 

YES  (75%) 

NO  (25%) 

SDR  2  Success 

YES  (90%) 

NO  (10%) 

MDA  Approval 
(Cost  Growth 
Check) 

YES  (65%) 

NO  (35%) 

In  reviewing  the  DoDI  5000.02  updated  in  2008  and  additional  guidance  memos,  several 
areas  were  identified  for  updates  in  the  model.  One  of  the  main  areas  of  change  was  the 
movement  of  PDR  prior  to  MS-B. 

When  consistent  with  Technology  Development  Phase  objectives,  associated  prototyping 
activity,  and  the  MDA-approved  Technology  Development  Strategy  (TDS),  the  PM  shall 
plan  a  PDR  before  MS-B.  PDR  planning  shall  be  reflected  in  the  TDS  and  shall  be 
conducted  for  the  candidate  design(s)  to  establish  the  allocated  baseline  (hardware, 
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software,  human/support  systems)  and  underlying  architectures  and  to  define  a  high- 
confidence  design  (OUSD/ATL  2008). 


PDR  in  ERAM  1.1  occurs  post  MS-B  as  does  System  Design  and  Demonstration  (SDD);  PDR  in 
ERAM  1 .2  has  PDR  activities  both  before  and  after  MS-B,  with  the  latter  optional.  SDD  is 
removed  in  ERAM  1.2  and  replaced  with  a  technology  development  identifier  pre  MS-B.  Within 
the  ExtendSim®  modeling  code,  the  PDR  activity  is  triggered  in  an  uncertainty  event  flow,  as 
shown  in  Figure  3 1 .  This  uncertainty  event  flow  is  an  artificial  way  to  introduce  some 
randomness  to  how  an  entity  progresses  through  the  model.  This  uncertainty  flow  continuously 
checks  random  numbers  to  a  probability  distribution.  If  the  number  falls  within  the  probability 
distribution,  the  event  proceeds  forward  in  the  process  flow.  In  ERAM  1.1,  since  the  PDR  was 
part  of  technology  development,  the  contract  association  changed  from  25%  contract  completion 
percentage.  For  PDR  this  percentage  will  be  80%.  The  trigger  for  these  events  is  an  uncertainty 
trigger  which  attempts  to  model  various  unpredictable  events  levied  on  the  program  office.  It 
occurs  about  every  30  days  based  on  a  random  generation  of  an  event.  It  could  be  a  funding  cut 
drill  or  some  other  activity  to  defend  their  acquisition  strategy.  This  is  an  out-of-cycle  activity 
levied  on  the  program  manager,  which  may  or  may  not  happen.  So  therefore,  this  “uncertainty 
event”  captured  this  unplanned  activity  path.  Figure  3 1  demonstrates  this  uncertainty  path  for 
the  PDR  in  the  contractor’s  swim  lane  which  will  be  moved  prior  to  MS-B.  It  checks  the 
progress  of  the  contract  with  contract  length  and  time.  If  the  uncertainty  event  levied  on  the 
program  office  occurs  at  a  time  where  80%  or  more  of  the  contract  has  progressed  sufficiently, 
the  PDR  is  triggered.  The  PDR  variable  is  set  which  kicks  off  the  activities  in  Figure  32. 


96 


Figure  31:  Move  PDR  to  Pre-B,  Add  SDR  (Uncertainty  Flow) 

Figure  32  occurs  in  the  main  acquisition  swim  lane.  After  the  PDR  event  is  triggered 

from  the  uncertainty  event  flow,  approximately  80%  of  the  technology  development  contract  is 
complete  and  the  SPO  prepares  for  a  down  selection  of  competitive  prototypes.  Additionally,  as 
indicated  in  Figure  31,  a  PDR  failure  will  result  in  a  rework  cycle  to  for  the  contractor  and  SPO 
to  make  corrections  and  then  go  back  for  approval.  This  describes  the  original  technology 
development  activity  prior  to  beginning  testing.  The  original  technology  development  had  lower 
levels  of  scrutiny.  However,  the  changes  in  the  DoDI  5000.02  place  a  PDR  toward  the  end  of 
technology  development  and  prior  to  MS-B.  “The  purpose  of  this  phase  is  to  reduce  technology 
risk,  determine  and  mature  the  appropriate  set  of  technologies  to  be  integrated  into  a  full  system, 
and  to  demonstrate  critical  technology  elements  (CTE)  on  prototypes.”  (OUSD/ATL  2008). 
Figure  32  describes  the  addition  of  the  SDR  and  PDR  event  flows  prior  to  MS-B.  It  also  adds 
the  Clinger-Cohen  Compliance  Assessment  and  approval  process  which  runs  in  parallel  to  those 
reviews  in  preparation  for  MS-B. 
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Figure  32:  Move  PDR  to  Pre-B,  Add  SDR  (Main  Flow) 

A  specific  requirement  for  space  systems  directed  by  OSD/AT&L  in  DTM  09-025,  Space 

Systems  Acquisition  Policy  (18  October  2010)  was  that  there  would  be  a  SDR  early  in 
acquisition  “During  the  technology  development  phase,  space  system  PMs  shall  conduct  an  SDR 
to  ensure  that  the  system’s  functional  baseline  is  established  and  that  the  system  has  a  reasonable 
expectation  of  satisfying  the  requirement  of  the  ICD  within  the  currently  allocated  budget  and 
schedule”  (Carter  2010).  Figure  33  shows  the  implementation  of  the  SDR  process  flow. 
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Figure  33:  SDR  Submodel 

For  SDR,  the  Space  Specific  sub-model  in  Figure  34  includes  the  IPA  support  of  the  Post 
SDR  assessment.  The  “What  IF”  path  was  removed  for  SDR  due  to  the  fact  that  all  SDRs 
include  an  IPA.  Where  there  is  a  mandatory  IPA  immediately  after  the  SDR,  Figure  35  describes 
that  activity’s  implementation.  This  illustrates  an  example  of  the  process  required  for  interacting 
between  SMC/XR  model  developers  and  the  authors  of  this  report. 


Figure  35  demonstrates  a  draft  implementation  of  the  “What  IF”  sub-model.  This  is  a 
small  portion  of  code  that  implements  some  of  the  activities  described  in  the  Space  Systems 
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Acquisition  Policy  DTM.  It  describes  additional  activities  to  be  associated  with  each  review  at 
the  MDA  level  in  addition  to  the  typical  milestone  reviews.  “The  MDA  shall  conduct  a  formal 
program  assessment  following  the  SDR  for  space  systems.  The  SDR  provides  an  opportunity  to 
assess  satisfaction  of  user  needs  through  functional  decomposition  and  traceability  of 
requirements  from  the  ICD  to  the  contractor’s  functional  baseline  and  system  specification.  An 
IPA  shall  be  provided  to  support  the  P-SDRA”  (Carter  2010).  The  “What  IF”  sub-model  checks 
to  determine  if  an  IPA  is  requested  by  the  MDA.  For  SDRs,  the  IPA  will  always  be  required  to 
support  the  MDA’s  post  SDR  assessment.  For  PDR  and  CDR,  the  MDA  can  decide  whether  or 
not  to  receive  support  from  an  IPA.  Hence,  the  “What  IF”  sub-model  performs  that  check 
following  the  particular  review.  If  the  MDA  does  not  approve,  there  is  a  single  rework  cycle  for 
the  SPO  to  make  corrections.  For  the  PDR  and  CDR  post  assessments,  the  “What  If’  sub-model 
is  included  as  a  check  to  see  if  the  MDA  requests  IPA  support. 


Figure  35:  "What  If"  Submodel 

MDA  approval  is  needed  after  the  affordability  assessment  if  the  cost  growth  is  greater 
than  25%  over  the  original  estimate. 

“If,  during  Technology  Development,  the  cost  estimate  upon  which  the  MDA  based  the 
Milestone  A  certification  increases  by  25  percent  or  more,  the  PM  shall  notify  the  MDA 
of  the  increase.  The  MDA  shall  again  consult  with  the  JROC  on  matters  related  to 
program  requirements  and  the  military  need(s)  of  the  system.  The  MDA  shall  determine 
whether  the  level  of  resources  required  to  develop  and  procure  the  system  remains 
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consistent  with  the  priority  level  assigned  by  the  JROC.  If  not,  the  MDA  may  rescind  the 
Milestone  A  approval  if  the  MDA  determines  that  such  action  is  in  the  interest  of  national 
defense”  (OUSD/ATL  2008). 

This  was  implemented  after  the  Affordability  Assessment  in  the  Pre  MS-B  activity.  Figure  36 
describes  how  the  DoDI  5000.02  text  was  turned  into  a  process  flow  diagram.  This  evolved  into 
an  implementation  shown  in  Figure  37.  Figure  38  shows  how  the  Cost  Growth  Check  sub-model 
was  added  into  the  main  flow  if  the  ExtendSim®  implementation 


Figure  36:  Notional  Cost  Growth  Process  Flow 
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Figure  38:  Cost  Growth  Check  in  Main  Flow 

Therefore,  the  above  changes  were  implemented  into  ERAM  1.2  as  a  portion  of  the 

collaboration  effort  with  SMC/XR.  The  DoDI  5000.02  guidance  was  checked  to  ensure  that  the 
independent  cost  estimate  (ICE)  activities  in  ERAM  1 .2  accurately  reflected  the  updates  to  the 
DoDI  5000.02.  The  structure  of  the  flow  did  not  change  in  the  ICE  portion.  The  Clinger-Cohen 
Act  Acceptance/Review  process  was  incorporated  as  a  significant  event.  Additionally,  the  IP  As 
and  Post-SDR/PDR/CDR  assessments  were  added  which  have  a  significant  impact  on  program 
timelines.  They  also  add  potential  exit  points  to  ERAM  1 .2  with  probabilities. 

ERAM  2.0 

ERAM  2.0  incorporated  the  global  variable  of  acquisition  capability  into  ERAM  1.2. 
Table  4  identifies  areas  where  an  ERAM  user  can  identify  the  appropriate  level  to  reflect  the 
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capabilities  of  the  SPO.  Appendix  B  documents  the  activities  selected  that  will  be  impacted  by 
this  global  variable.  The  global  variable  has  levels  from  0  to  4  which  determine  the  triangular 
distribution  impact  for  the  timelines  of  the  discrete  events.  Figure  39  shows  a  notional  view  of 
how  the  distributions  will  be  impacted  based  on  experience  and  staffing  levels.  The  “y”  axis 
shows  the  probabilities  while  the  “x”  axis  shows  the  timelines  in  number  of  days.  It  suggests 
lower  capability  skews  events  longer,  while  higher  capability  skews  the  event  to  be  completed  in 
a  more  timely  fashion.  Level  2  is  considered  a  baseline  level  which  doesn’t  skew  activities  as 
modeled  in  any  direction. 
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Personnel  Experience  Global  Variable  Inputs 


Figure  39:  Notional  Triangular  Distribution  Timeline  Impacts 

ERAM2.1 

In  order  to  implement  the  JCIDS  process  in  ERAM  1.2,  several  changes  and  additions 
were  required.  The  Microsoft  VISIO  diagram  that  was  reviewed  with  the  requirements  and 
acquisition  personnel  was  compared  with  ERAM  1.2  to  determine  insertion  points  and  changes. 
Table  14  outlines  the  activities  recommended  for  implementation  in  ERAM  2.1. 
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Table  15:  JCIDS  Model  Changes 


Model  Changes 

1 

Add  JCIDS  Initiation 

2 

Add/Update  JCIDS  Concept  Review  Process 

3 

Add  Development  Planning  Initiation 

4 

Add  CCTD  Development  Process 

5 

Add/Update  MDD  Process 

6 

Update  AoA  Process 

7 

Add  JUON  Process 

Figures  40  and  41  demonstrate  how  the  model  design  was  recommended  to  the 
Aerospace  software  engineers  to  fully  implement  the  pre-MDD  JCIDS  activities.  In  Figure  40, 
the  first  column  of  activities  represents  potential  sources  of  “ideas”  for  capabilities  and/or 
concepts  that  are  injected  into  the  “AFSPC  IPP  &  ATC”.  The  “AFSPC  IPP  &  ATD”  at  a  given 
point  in  time  will  prioritize  and  select  which  capabilities  to  pursue  further.  The  first  step  is  to 
compare  the  capability  against  existing  ICDs.  If  one  exists,  it  by-passes  the  JROC  ICD  approval 
flow.  However,  if  it  requires  a  new  ICD,  depending  on  if  the  capability  or  technology  is  a  paper 
concept,  a  prototype/ JCTD  or  an  ATD  will  determine  which  activity  path  it  follows.  Translating 
this  to  code,  the  “For  existing  Program”  icon  in  Figure  41  becomes  a  user  selected  “pull-down 
menu”  to  choose  ATD,  Paper  Concepts,  or  others. 
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Figure  40:  JCIDS  Initiation 


Figure  41:  JCIDS  Initiation  ExtendSim  Implementation 

Figures  42  through  44  describe  the  concepts  as  they  proceed  through  the  CBA  process. 

The  concept  then  goes  through  a  DOTMLPF  decision.  It  follows  separate  paths  based  upon 
sponsorship  from  either  AFSPC  or  USTRATCOM.  If  it  is  sponsored  by  USTRATCOM,  it 
bypasses  the  Air  Force  specific  processes.  The  implementation  included  reuse  of  existing  code 
from  the  Joint  Interest  pre  MS-B  sub-model  shown  in  Figure  43.  A  significant  difference  is  that 
the  USTRATCOM  sponsored  concept  will  bypass  the  AFROC  and  insert  directly  into  the  middle 
of  the  Joint  Interest  sub-model.  The  Functional  Review  Board  in  Figure  41  occurs  in  the  Joint 
Interest  sub-model. 
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Figure  42:  JCIDS  Concept  Review  Process 


Figure  43:  JCIDS  Concept  Review  Process  ExtendSim  Implementation  Part  1 


Figure  44:  JCIDS  Concept  Review  Process  ExtendSim  Implementation  Part  2 
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Figures  45  and  46  describe  the  initiation  of  the  DP  process  initiated  after  an  ICD  is 
approved.  This  AFSPC  initiated  processes  sends  the  DP  Request  to  SMC/XR  or  SPO  for  an 
existing  program  of  record.  This  is  then  reviewed  and  processed  through  the  AFSPC  led 
prioritization  process. 


Figure  45:  Development  Planning  Initiation 


Figure  46:  Development  Planning  Initiation  ExtendSim  Implementation 
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Figures  47  and  48  describe  the  various  steps  that  SMC/XR  or  a  SPO  XR  branch 
accomplishes  as  they  complete  the  CCTD  effort.  The  CCTD  process  owner  accomplishes  five 
phases,  as  shown  in  Figure  47.  In  the  implementation  in  Figure  48,  the  five  phases  are  executed 
sequentially  as  a  substitute  for  the  DP  block.  This  is  in  concert  with  the  intent  of  the  designer. 
However,  as  the  design  team  and  the  software  engineers  met,  the  5  CCTD  phases  were 
aggregated  back  up  to  a  single  DP  activity  block. 


Figure  47:  CCTD  Development  Process 
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Figure  48:  CCTD  Development  Process  ExtendSim  Implementation 

Figures  49  and  50  describe  how  the  MDD  and  approval  were  modeled.  The  ADM  and 

MDD  brief  submitted  to  the  MDD  result  in  a  decision  which  either  terminates  the  program  or 
sends  it  further  down  the  process  to  begin  an  AoA.  The  implementation  in  Figure  50  reused 
existing  ERAM  1.1  code  and  clarified  the  processes  to  include  the  AFRB. 


Figure  49:  MDD  Process 


Figure  50:  MDD  Process  ExtendSim  Implementation 

Figures  51  through  53  show  the  AoA  design  and  implementation  through  the  point  the 

program  is  inserted  into  the  acquisition  process  based  on  the  ADM.  Existing  ERAM  1 . 1  code 
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was  reused  which  captured  the  detail  of  the  AoA  process  through  developing  courses  of  action. 
In  Figure  53,  the  different  insertion  points  after  a  preferred  concept  is  selected  and  the  3  way 
branch  either  continues  in  the  pre  MS-A,  or  jumps  to  pre  MS-B  or  pre  MS-C. 


Figure  51:  Analysis  of  Alternatives 


Figure  52:  Analysis  of  Alternatives  ExtendSim  Implementation  Part  1 
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Figure  53:  Analysis  of  Alternatives  ExtendSim  Implementation  Part  2 

Figures  54  through  57  show  the  separate  JUON  process.  If  the  user  selected  menu  in 

Figure  40  and  41  selects  JUON,  it  follows  the  new  path.  This  path  ends  at  a  fielded  capability  if 
it  passes  all  of  the  review  processes.  The  final  decision  results  in  a  branching  that  selects 
whether  or  not  it  will  become  an  enduring  program  and  go  into  the  official  process,  be  disposed 
of  or  sustained  exclusively  in  the  theater  of  operations. 


Figure  54:  JUON  Process  Part  1 


Figure  55:  JUON  Process  Part  1  ExtendSim  Implementation 
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Figure  56:  JUON  Process  Part  2 


Figure  57:  JUON  Process  Part  2  ExtendSim  Implementation 

4.5  Phase  5:  Document  and  Report 

At  the  conclusion  of  these  interviews,  modeling  updates,  document  reviews,  etc,  the 
research  team  captured  the  data  and  documented  the  findings.  These  findings  were  also 
presented  orally  on  multiple  occasions  for  academic  purposes  and  to  satisfy  the  requirements  of 
the  sponsoring  organization.  A  final  review  was  conducted  with  the  Aerospace  software 
engineers  to  ensure  they  had  obtained  the  required  data  for  completing  the  model 
implementation. 
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V.  Conclusion  &  Recommendations 

5.1  Conclusion 

Unfortunately,  defense  space  acquisitions  have  earned  a  reputation  of  being  notoriously 
over  budget  and  behind  schedule.  Not  surprisingly,  Congressional  and  senior  DoD  leadership 
have  recently  focused  efforts  on  identifying  the  root  cause  in  order  to  improve  program 
execution.  As  previously  reported  by  the  GAO,  this  research  also  found  that  many  early  space 
program  decisions  have  been  based  on  insufficient  and/or  inaccurate  data  with  regards  to 
schedule  and  cost  estimates  as  well  as  technology  maturity.  In  an  abstract  relationship,  a  DoD 
capability  gap  exists  for  its  ability  to  properly  estimate  and  execute  the  cost  and  schedule  to 
deliver  new  capabilities. 

In  an  effort  to  provide  a  solution  to  this  capability  gap,  policy  and  statutory  guidance  has 
been  updated  in  the  past  several  years  emphasizing  the  importance  of  early  systems  engineering, 
DP,  and  assessing  technology  maturity  in  advance  of  MS  B.  Tools  such  as  ERAM,  RAMP, 
ADDM  and  APM  continue  to  gain  attention  as  they  provide  insight  into  the  acquisition  system 
and  help  characterize  the  various  decisions  points  and  activity  timelines.  Through  the  process  of 
updating  ERAM  for  space  acquisitions,  the  following  are  several  valuable  observations  for 
potential  contributions  to  the  current  challenges  faced  in  space  capability  development: 

1)  ERAM  provides  a  comprehensive  early  schedule  estimate  based  upon  existing 
government  policy  and  instructions,  a  valuable  piece  of  data  currently  missing.  For  a 
given  concept,  this  data  should  be  captured  in  the  CCTD  and  properly  assessed  in  the 
AoA  prior  to  a  materiel  solution  being  selected. 

2)  ERAM  provides  insight  into  the  extraordinarily  complex  process  of  developing  space 
capabilities.  A  single  model  that  comprehensively  ties  the  multitude  of  processes  and 


organizations  outlined  by  DoD  and  AF  policy,  instructions,  and  guides  has  the 
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potential  reducing  and  simplifying  the  process  into  an  understandable  and 
manageable  process  for  requirements  and  acquisition  personnel. 

3)  ERAM  provides  detailed  context  to  the  well  known,  but  too  often  in  comprehensible, 
DAU  acquisition  wall  chart.  The  DAU  wall  chart  provides  a  high  level  picture  of  the 
activities,  but,  lacks  the  detail  to  be  an  effective  management  tool.  ERAM’s  activity 
diagram  identifies  the  sequence  of  activities/decisions,  the  organizations  responsible 
for  them  and  estimated  timelines  for  the  activities,  resulting  in  a  management  tool  that 
empowers  PMs  to  more  effective. 

4)  ERAM  could  become  the  backbone  of  a  training  program  for  DoD  and  industry  PMs 
providing  detailed  insight  to  current  space  acquisition  policy  and  processes;  a  type  of 
program  management  “flight  simulator”. 

5)  ERAM  is  an  asset  available  for  SMC/XR  to  utilize  during  DP  efforts  (developing 
CCTDs)  for  AFSPC  with  regards  to  “revolutionary”  capabilities  while  existing  SPO 
XR  shops  can  leverage  SMC/XR  CCTD  expertise  and  ERAM  for  AFSPC  directed 
“evolutionary”  capabilities  DP. 

At  the  beginning  of  this  research  effort,  three  objectives  were  established:  update  ERAM 
for  space  capability  development  in  ExtendSim  modeling  software,  verify  and  validate  the  model 
and  lastly,  provide  recommendations  for  further  research  efforts  (if  required). 

The  first  objective  of  updating  ERAM  in  ExtendSim  for  space  acquisitions  was  achieved; 
however,  additional  research  remains  in  order  to  reach  a  version  capable  of  providing  usable  data 
for  CCTDs.  Research  efforts  found  that  there  is  no  single  “correct”  process  that  everyone  agrees 
upon  or  one  that  can  be  designed  from  published  guidance.  The  myriad  of  policy  memos, 
instructions  and  guides  are  difficult  to  tie  together  into  a  single  activity  diagram;  rather,  they 
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provide  a  framework  that  must  be  navigated  uniquely  for  every  program.  Despite  these 
challenges,  a  generally  accepted  framework  has  been  designed  and  implemented  as  ERAM  2. 1 . 
Nevertheless,  the  authors  used  their  own  expertise  and  knowledge  gained  during  this  research  to 
suggest  notional  data  that  can  be  refined  later.  Rather,  the  data  generated  by  the  authors  is  of 
high  quality  with  reliable  resources  and  meets  the  triangulation  method  of  validation  (Patton  and 
Denzin,  qtd.  in  Scharch,  et  al.  2011).  This  was  accomplished  through  interviews  with  SMEs 
familiar  with  space  capability  development  pre  MS-A,  analysis  of  existing  policy  and  guidance 
governing  capability  development  as  well  as  comparing  the  model  to  current,  real  world  pre  MS- 
A  space  concepts.  Objective  three  was  met  and  documented  in  the  “Recommendations”  portion 
of  this  report. 

5.2  Recommendations 

With  all  of  the  uncertainty  and  variability  within  the  DoD  acquisition  process,  there  will 
always  be  modeling  opportunities  to  improve  the  fidelity.  As  Statistician  George  Box  stated, 
“All  models  are  wrong,  but  some  are  useful.”  (Box  Vol  71)  The  following  are  several  research 
topics  that  would  continue  to  increase  the  utility  of  ERAM  for  space  capability  development: 

1)  PPB&E  Activities/Decisions:  More  detailed  research  into  modeling  PPB&E  Activities  would 
add  realism  to  the  model.  This  calendar  driven-process  drives  the  funding  available  to  program 
managers  at  various  stages.  Spring  execution  reviews  and  other  events  have  the  potential  to 
remove  funding  from  the  program.  This  process  exists  in  ERAM,  but  would  be  improved  with 
further  research. 

2)  T&E  Activities/Decisions:  Test  and  Evaluation  as  a  significant  portion  of  the  acquisition 
community  is  a  good  candidate  for  research  emphasis  on  future  spirals  of  ERAM.  It  is  included 
in  the  current  version  of  ERAM  but  at  a  high  level  of  abstraction.  Coordination  with  the 
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Operational  Test  community  is  essential  early  in  acquisition  with  Test  and  Evaluation  Master 
Plans  required  at  Milestone  reviews. 

3)  ERAM  Maintenance:  In  general  terms,  further  research  in  probability  distributions  for 
activities  and  decision  points  would  increase  the  fidelity  of  the  model.  Larger  sampling  of 
subject  matter  experts  will  further  refine  the  spans  of  the  distribution.  Keeping  up  with  the 
frequent  policy  changes  will  also  add  to  the  maintenance  of  ERAM. 

4)  Improve  acquisition  modeling  efforts  between  SAF/AQ,  AFMC,  and  the  XR  shops  at  ASC, 
ESC,  AAC  and  SMC  to  reduce  duplication  of  effort  and  maximize  synergies  between  various 
models. 

5)  Analyze  how  the  cost  of  early  SE  and  increased  quantitative  analysis  impact  the  overall 
performance  (cost  and  schedule)  of  programs  compared  to  previous  programs  with  little  early  SE 
(utility  analysis).  These  observations  could  then  be  folded  into  future  versions  of  ERAM. 

6)  Further  development  and  analysis  of  the  rapid  acquisition  process  (JUONs  as  well  as  other 
techniques). 

In  the  process  of  identifying  a  comprehensive  activity  diagram  that  ties  S&T,  AFSPC 
capability  gap  analysis,  JCIDS,  DP  and  eventually  the  acquisition  of  a  materiel  solution,  several 
general  challenge  areas  with  regards  to  space  capability  development  are  discussed: 

Challenge  #1:  Inadequate  Early  Systems  Engineering.  Lack  of  early  systems  engineering 
results  in  a  weak  technical  foundation  for  the  system  to  be  built  upon  and  results  in  an  increased 
number  of  risks  and  “unknown  unknowns”.  Too  often,  the  hard  questions  are  pushed  off  to  be 
addressed  later  in  the  program,  resulting  in  costly  schedule  delays. 

Proposed  Solution:  Improve  early  systems  engineering  analysis  prior  to  MDD  and  MS- A 
approval.  Early  systems  engineering  will  help  identify  and  assess  the  risks  associated  with  these 
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questions  before  major  program  decisions  occur,  preventing  more  costly  impacts  further  into  the 
program.  Recent  guidance  has  directed  this;  however,  implementation  is  still  in  progress.  It  is 
recommend  that  to  overcome  the  challenges  that  remain  1)  assign  responsibility  to  a  single 
organization,  and  2)  find/hire  technically  competent  systems  engineers  to  develop  the  space 
architecture,  and  3)  choose  a  single  systems  engineering  software  suite  that  is  required  for  use  by 
all  services. 

Challenge  #2:  Failure  of  the  Government  to  Assess  Technology  Maturity.  Poor  technology 
maturity  assessments  prior  to  contract  award  results  in  costly  schedule  delays. 

Proposed  Solution:  To  accurately  assess  technology  maturity  levels,  the  government  must 
improve  its  technical  knowledge  base  and  establish  improved  evaluation  methods.  Rather  than 
accept  ideas  at  face  value  as  “good”,  proving  the  feasibility  must  be  required.  Improving  the 
process  for  AFRL,  universities  and  industry  partners  to  prove  military  utility  of  hardware  and 
software  technologies  in  advance  of  contract  award  will  likely  significantly  improve  contract 
performance  post  Milestone  B. 

Challenge  #3:  Improper  Distribution  of  Personnel.  Developing  new  space  capabilities  and 
sustaining  existing  capabilities  is  indeed  a  challenging  endeavor.  In  response,  in  the  opinion  of 
the  authors,  the  number  of  personnel  supporting  HHQs  appears  to  continue  to  increase,  resulting 
in  significant  manpower  (and  associated  costs)  charged  with  developing  new  and  improved 
policy  and  instructions.  At  the  product  centers,  very  few  PMs  can  piece  the  policy  architecture 
together  or  have  the  time  to  in  order  to  make  sense  of  how  the  multi-organizational  process  is 
designed  to  work. 

Proposed  Solution:  First,  reduce  and  consolidate  the  number  of  guidance  documents.  Ensure 
the  documents  are  consistent  with  one  another  and  establish  a  coherent  process  flow  that  doesn’t 
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require  a  genius  to  understand.  After  this  is  accomplished,  second,  reallocate  the  manpower 
positions  supporting  HHQ  to  support  the  personnel  at  the  Centers.  Third,  with  a  portion  of  these 
resources,  establish  a  rigorous  formal  training  and  evaluation  program  that  qualifies  personnel 
assigned  to  key  positions.  Current  DAU  training  is  insufficient;  just  as  operators  have  formal 
training  to  learn  how  to  operate  their  weapon  system  and  understand  technical  orders,  personnel 
involved  in  the  acquisition  system  should  be  required  to  understand  the  acquisition  system  and 
associated  instructions.  In  addition  to  initial  training  requirements,  establish  recurring  training 
and  evaluation  programs  that  re-enforces  and  sustains  the  knowledge  base.  Fourth,  with  the 
remaining  reallocated  resources  from  HHQs,  increase  the  number  of  technical  positions  at  the 
Centers.  It’s  the  Centers  that  are  responsible  for  early  systems  engineering,  CCTD  documents 
and  executing  programs  on  schedule  and  within  cost  and  it’s  the  Centers  that  have  historically 
struggled. 

Challenge  #4:  Lack  of  Process  Discipline  &  Training.  Individual  organizations  (AFSPC, 
SMC,  AFRL,  SAF,  OSD,  AFOTEC)  all  have  their  individual  processes  and  procedures  for 
fulfilling  their  responsibilities.  With  senior  leadership  within  these  organizations  turning  over 
approximately  every  two  years,  there  is  a  good  chance  that  those  processes/products  change 
every  two  years.  This  is  not  surprising  since  there  are  so  many  issues  throughout  the  entire 
system.  The  intentions  of  these  changes  are  positive,  but  the  results  have  far  reaching 
implications  that  aren’t  properly  communicated/trained  and  difficult  to  absorb.  This  impairs  the 
ability  for  other  organizations  dependent  on  that  organization  to  understand  how  the  process 
works.  Additionally,  it  makes  it  extremely  difficult  to  establish  value  added  training  curriculum 
since  the  material  would  become  outdated  more  quickly  than  it  could  be  approved. 
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Proposed  Solution:  Establish  organizational  process  discipline.  Should  organizations  choose  to 
change  processes  that  impacts  other  organizations  in  the  “system”,  then  there  should  be  a  formal 
change  process.  Part  of  the  process  would  require  outreach  training  to  other  organizations  that 
rely  on  them  for  their  own  success/progress. 

Challenge  #5:  Poor  6X  Training  &  Evaluation  Program.  Acquisition  personnel  are  required 
by  law  to  take  training  courses  offered  by  the  Defense  Acquisition  University  and  maintain 
Continuous  Learning  Points.  This  training  is  insufficient  and  only  scratches  the  surface  with 
regards  to  how  the  acquisition  system  truly  functions. 

Proposed  Solution:  DoD  and  SAF  need  to  establish  a  formal  training  and  evaluation  program 
for  requirements  and  acquisition  personnel.  AETC  is  responsible  for  operational  system 
qualification  training  for  both  intelligence,  space  and  aerospace  system  operations.  Additionally, 
the  operational  units  have  established  formal  evaluations  to  ensure  the  qualification  and 
proficiency  of  the  operators.  The  requirements  and  acquisition  system  is  certainly  different, 
however,  establishing  formal  training  and  evaluation  programs  could  enforce  an  in  depth 
understanding  of  the  processes  and  pay  significant  dividends  to  ensure  personnel  are 
knowledgeable  and  qualified.  Additionally,  this  may  serve  as  a  forcing  function  to  HHQ  to 
reduce  and  simplify  the  policy  guidance  and  instill  process  discipline  throughout  various 
organizations  involved. 

Challenge  #6:  Process  Disconnects  between  MAJCOM  &  AFRL.  The  S&T  process  for 
focusing  defense  resources  on  maturing  technologies  for  desired  capabilities  is  disjointed  from 
AFSPC  and  the  JCIDS  process. 

Proposed  Solution:  AFSPC  needs  to  improve  its  ability  to  communicate  technology 
development  requirements  to  AFRL.  AFRL  needs  to  improve  their  responsiveness  to  AFSPC  on 
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these  efforts.  At  the  same  time,  AFRL  must  also  continue  to  provide  AFSPC  with  ideas  on 
emerging  military  technology  applications  so  the  classified  capability  development  strategies  can 
continue  to  evolve. 

In  conclusion,  failure  to  for  the  Department  of  Defense  to  take  actions  to  improve  an 
inadequate  space  acquisition  system  threatens  the  United  States  supremacy  in  space  as  well  as  its 
national  security.  Adversaries  are  increasing  their  ability  to  develop  technologies  and  field  them 
at  an  alarming  rate,  closing  the  technology  gap  America  has  become  accustomed  to.  From 
President  Obama’s  National  Security  Strategy,  “We  will  invest  in  the  research  and  development 
of  next-generation  space  technologies  and  capabilities  that  benefit  our  commercial,  civil, 
scientific  exploration,  and  national  security  communities,  in  order  to  maintain  the  viability  of 
space  for  future  generations.”  In  a  fiscally  constrained  environment,  as  the  service  responsible 
for  space  capabilities,  the  United  States  Air  Force  must  tackle  this  challenge.  ERAM  is  not  the 
solution  to  the  major  problem,  but  can  provide  valuable  context  to  the  extraordinarily  complex 
capability  development  system  that  requires  significant  reform. 
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APPENDIX  A.  List  of  Acronyms 


ACAT 

ACE 

ACPO 

ADM 

ADDM 

AIP 

APM 

ATD 

ASC 

AAC 

ACPO 

AFIT 

AFMC 

AFMCI 

AFPC 

AFPD 

AFRB 

AFROC 

AFRL 

AFSPC 

AFSPCI 

AMA 

ATC 

ASD/NII 

CDD 

CBA 

CPD 

CRET 

COCOM 

C2 

CCTD 

CDC 

CO 

CAPE 

CTE 

DAB 

DAU 

DoD 

DoDI 

DP 

DTM 


Acquisition  Category 

Acquisition  Center  of  Excellence 

Acquisition  Chief  Process  Office 

Acquisition  Decision  Memorandum 

Acquisition  Document  Development  and  Management 

Acquisition  Improvement  Plan 

Acquisition  Process  Model 

Advanced  Technology  Demonstration 

Aeronautical  Systems  Center 

Air  Armament  Center 

Acquisition  Chief  Process  Office 

Air  Force  Institute  of  Technology 

Air  Force  Materiel  Command 

Air  Force  Materiel  Command  Instruction 

Air  Force  Personnel  Command 

Air  Force  Policy  Document 

Air  Force  Review  Board 

Air  Force  Requirements  Oversight  Council 

Air  Force  Research  Laboratory 

Air  Force  Space  Command 

Air  Force  Space  Command  Instruction 

Analysis  of  the  Materiel  Alternatives 

Applied  Technology  Council 

Assistant  Secretary  of  Defense  for  Networks  Information  &  Integration 

Capabilities  Development  Document 

Capabilities-Based  Assessment 

Capability  Production  Document 

Center  for  Re-engineering  Technology 

Combatant  Command 

Command  and  Control 

Concept  Characterization  and  Technical  Description 
Concept  Development  Center 
Contracting  Officer 

Cost  Assessment  &  Program  Evaluation 
Critical  Technology  Element 
Defense  Acquisition  Board 
Defense  Acquisition  University 
Department  of  Defense 
Department  of  Defense  Instruction 
Development  Planning 
Directive  Type  Memorandum 
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ESC 

Electronic  Systems  Center 

ECP 

Engineering  Change  Proposal 

ERAM 

Enterprise  Requirements  and  Acquisition  Model 

FCB 

Functional  Capabilities  Board 

FFRDC 

Federally  Funded  Research  and  Development  Center 

FSA 

Functional  Solutions  Analysis 

GAO 

Government  Accountability  Office 

GRP 

Group  Research  Project 

HPT 

High  Performance  Team 

HHQ 

Higher  Headquarters 

IED 

Improvised  Explosive  Device 

ICE 

Independent  Cost  Estimate 

IPA 

Independent  Program  Assessment 

IRAD 

Independent  Research  and  Development 

IRSS 

Information  and  Resource  Support  System 

IT 

Information  Technology 

ICD 

Initial  Capability  Document 

IPP 

Integrated  Planning  Process 

ISR 

Intelligence,  Surveillance  and  Reconnaissance 

JCB 

Joint  Capabilities  Board 

JCA 

Joint  Capability  Area 

JCIDS 

Joint  Capability  Integration  and  Development  System 

JCTD 

Joint  Capability  Technology  Demonstration 

JPD 

Joint  Potential  Designator 

JROC 

Joint  Requirements  Oversight  Council 

JUON 

Joint  Urgent  Operational  Need 

KPP 

Key  Performance  Parameter 

KM/DS 

Knowledge  Management/Decision  Support 

MAJCOM 

Major  Command 

MDAP 

Major  Defense  Acquisition  Program 

MDD 

Materiel  Development  Decision 

MS 

Milestone 

MDA 

Milestone  Decision  Authority 

MUA 

Military  Utility  Assessment 

NRO 

National  Reconnaissance  Office 

OSD/AT&L 

Office  of  the  Secretary  of  Defense,  Acquisition,  Technology  &  Logistics 

ORD 

Operational  Requirements  Document 

PDR 

Preliminary  Design  Review 

P-SDRA 

Pre-System  Design  Review  Assessment 

PE 

Program  Element 

PEO 

Program  Executive  Officer 

PM&AE 

Program  Management  &  Acquisition  Excellence 

PM 

Program  Manager 
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POM 

PPB&E 

RAMP 

RSR 

RMD 

S&T 

SAF/AECO 

SAF/AQ 

SAF/AQRE 

SCF 

SMC/XR 

SMC 

SRP 

SSAP 

SMP 

SME 

SDD 

SDR 

SPO 

TDS 

TRA 

TRL 

TSAT 

USD/AT&L 

USTRATCOM 

WSARA 

WBS 


Program  Objectives  Memorandum 
Program,  Planning,  Budget  &  Execution 
Requirements  and  Management  Plan 
Requirements  Strategy  Review 
Resource  Management  Decision 
Science  &  Technology 

Secretary  of  the  Air  Force,  Acquisition  Excellence  Change  Office 
Secretary  of  the  Air  Force,  Acquisitions 

Secretary  of  the  Air  Force,  Acquisitions  Engineering  &  Technical 

Management  Division 

Service  Core  Functions 

SMC  Development  Planning  Division 

Space  and  Missile  Systems  Center 

Space  Re-capitalization  Plan 

Space  Systems  Acquisition  Policy 

Strategic  Master  Plan 

Subject  Matter  Expert 

System  Design  &  Demonstration 

System  Design  Review 

System(s)  Program  Office 

Technology  Development  Strategy 

Technology  Readiness  Assessment 

Technology  Readiness  Level 

Transformational  Satellite  Communications  System 

Under  Secretary  of  Defense,  Acquisition,  Technology  &  Logistics 

United  States  Strategic  Command 

Weapon  System  Acquisition  Reform  Act 

Work  Breakdown  Structure 
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APPENDIX  B.  K  RAM  2.0  Activities  Impacted  by  Capability  Matrix 


Delay  for  protest 

Delay  for  protest  review  PreB 

Delay  for  protest  review  PreC 

EOA  rework  and  delay 

MAJCOM  approval  PreA 

AFROC  Preparations  PreA 

AFROC  Preparations  independent  document  PreA 

AFROC  Preparation  joint  interest  PreA 

AFROC  Document  Review  joint  interest  PreA 

FCB  and  JCB  joint  interest  PreA 

AFROC  Preparations  independent  document  PreB 

AFROC  Preparations  joint  integration  PreB 

Final  AFROC  approval  joint  integrations  PreB 

AFROC  Preparations  joint  interest  PreB 

AFROC  Document  Review  joint  interest  PreB 

FCB  and  JCB  and  JROC  joint  interest  PreB 

AFROC  Preparations  independent  document  PreC 

AFROC  Preparations  joint  integration  PreC 

Final  AFROC  Approval  joint  integration  PreC 

AFROC  Preparations  joint  interest  PreC 

AFROC  Document  Review  joint  interest  PreC 

FCB  and  JCB  and  JROC  joint  interest  PreC 

MAJCOM  Approval  independent  document  PreA 

MAJCOM  A  Fetters  and  Coordination  and  Concur 

MAJCOM  A8  and  RSR 

MAJCOM  Approval  joint  integrations  PreA 

MAJCOM  Approval  Fater  joint  integration  PreA 

MAJCOM  Approval  joint  interest  PreA 

MAJCOM  Approval  independent  document  PreB 

MAJCOM  A  Fetter  and  Coordinate  and  Concur  PreB 

MAJCOM  A8  and  RSR 

MAJCOM  Approval  joint  integration  PreB 

MAJCOM  Approval  Fater  joint  integration  PreB 

MAJCOM  Approval  joint  interest  PreB 

MAJCOM  A  Fetter  Coordination  and  Concur  PreC 

MAJCOM  A8  and  RSR  PreC 

MAJCOM  Approval  Independent  document  PreC 

MAJCOM  Approval  joint  integration  PreC 

MAJCOM  Approval  Fater  joint  integration  PreC 

MAJCOM  Approval  joint  interest  PreC 

SVR 

DRR 

SRR 

MDA  Milestone  Approval  PreB 
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MPA  Milestone  Approval  PreC 

MPA  Milestone  A  Approval _ 

RFP  Release  and  Source  Sel  Pre-B 
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APPENDIX  C.  Sample  Interview  Questions 


1)  Describe  your  roles  and  responsibilities  in  the  space  acquisition  processes. 

2)  What  space  programs  do  you  have  experience  with? 

3)  Describe  what  processes  you’ve  been  involved  with  regards  to  space  capability 
development  (i.e.  gap  analysis,  S&T,  JCIDS  documents,  DP,  and/or  acquisitions)  and 
with  which  program 

4)  Describe  the  specific  activities  and  decision  points  for  the  processes  you’ve  been 
involved  in  from  question  3. 

5)  What  reviews  and  documentation  were  required  for  your  program? 

6)  In  your  opinion,  was  there  a  clearly  identified  sponsor  (i.e.  owner)  for  the  documentation 
required  for  the  reviews?  If  not,  please  describe  the  discrepancy. 

7)  What  was  the  timeframe  for  approval  on  the  required  documentation?  Which 
organizations  were  required  for  coordination?  What  obstacles  needed  to  be  overcome? 

8)  Have  you  experienced  delays  with  your  program?  If  so,  what  would  you  say  was  the 
primary  driver  of  that  delay? 

9)  When  did  the  various  reviews  and  other  meetings  occur  in  your  program?  Who  was 
required  to  attend?  What  were  the  challenges  that  arose?  How  did  you  overcome  those 
challenges? 

10)  If  applicable,  what  waivers  were  applicable  for  moving  your  program  forward?  How  did 
you  obtain  approval  for  those  waivers? 

1 1)  In  the  future,  as  a  framework  of  the  activities  from  capability  gap  analysis  through 
Materiel  Development  Decision  is  developed,  would  you  be  willing  to  review  it  and 
provide  feedback  for  accuracy? 

12)  In  reference  to  the  previous  question,  would  you  be  willing  to  provide  data  on  the 
durations  of  the  activities  and  probabilities  for  decision  points? 

13)  Do  you  have  any  specific  persons  we  should  talk  to  at  AFSPC,  SAF/AQ,  USD/AT&L,  or 
other  agencies? 
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14)  Do  you  have  anything  else  you  would  like  us  to  consider  with  regards  to  modeling  and 
reporting  on  the  pre  Milestone-B  space  acquisition  activities? 
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